IaaS Performance and Value Analysis
A study of performance among the Top 14 public cloud infrastructure providers.
Download The Report

CloudPack Test Suite

What is CloudPack?
Performance is becoming a major force of innovation across enterprises and Infrastructure as a Service (IaaS) providers of all sizes.

Today IaaS providers offers multiple service packages to end customers, but there is still no efficient way to quantify which IaaS provider offer the best suitable package for the specific user workload. CloudPack enables that.

The Cloud500, a subgroup of the Cloud Advisory Council, created a suite of tests and developed a metric to enable comparability among Cloud Providers different packages offering in a unified measurable method, 'apples to apples' comparison.

The Current Cloud500 Test suite includes real-life workloads testing focused around System, RAM, and CPU.

CloudPack Test Suite

7-Zip Compression

Tests the performance of the virtual CPU by measuring the millions of instructions per second (MIPS) that it can handle when compressing a file.

LAME MP3 Audio Encoding

Tests how long it takes to encode a WAV file to MP3 format using LAME, an MP3 encoder.

x264 Video Encoding

Converts video from MPEG2 to MPEG4

Geekbench 3 Single Core - Integer Operations

CPU Integer test suite from Geekbench 3

AES Encryption

The AES workload encrypts a generated text string using the advanced encryption standard (AES). AES is used in security tools such as SSL, IPsec, and GPG. Geekbench uses the AES-NI instructions when they are available. When the AES-NI instructions are not available, Geekbench uses its own software AES implementation.

Twofish

The Twofish workload also encrypts a text string, but it uses the Twofish algorithm. Twofish is from the family of encryption algorithms known as "Feistel ciphers." It is included in the OpenPGP standard.

SHA1

SHA1 is a cryptographic hash algorithm: given a binary input it generates a "hash" or "digest" of the input. SHA1 is designed so that the hash may be computed quickly, but it is difficult to find a string that generates a given hash. SHA1 may be used, for example, to encrypt passwords by storing the hash instead of the password text. The SHA1 workload uses a text string as input.

SHA2

SHA2 solves the same problem as SHA1, but is more secure: SHA1 has a known vulnerability to "collision attacks." Although these attacks are still impractical and SHA1 is still widely used, it is being gradually replaced by SHA2.

BZip2 Compression and Decompression

BZip2 is a compression algorithm. The BZip2 workloads compress and decompress an ebook formatted using HTML. Geekbench 3 uses bzlib version 1.0.6 in the BZip2 workloads.

JPEG Compression and Decompression

The JPEG workloads compress and decompress one digital image using lossy JPEG format. The workloads use libjpeg version 6b.

PNG Compression and Decompression

The PNG workloads also compress and decompress a digital image, but they do so using the PNG format. The workloads use libpng 1.6.2.

Sobel

The "Sobel operator" is used in image processing for finding edges in images. The Sobel workload uses the same input image as the JPEG and PNG workloads.

Lua

Lua is lightweight scripting language. The Lua workload is similar to the code used to display Geekbench results in the Geekbench Browser.

Djikstra

The Dijkstra workload computes driving directions between a sequence of destinations. Similar techniques are used by AIs to compute paths in games and by network routers to route computer network traffic.

Geekbench 3 Single Core - Floating Point Operations

CPU Floating point test suite from Geekbench 3

Black Scholes

The Black-Scholes equation is used to model option prices on financial markets. The Black-Scholes workload computes the Black-Scholes formula: a special case solution of the Black-Scholes equation for European call and put options.

Mandelbrot

The Mandelbrot set is a fractal. It is a useful floating point workload because it has a low memory bandwidth requirement.

Sharpen Image

The sharpen image workload uses a standard image sharpening technique similar to those found in Photoshop or Gimp.

Blur Image

Image blurring is also found in tools such as Photoshop. In Geekbench 3, the blur image workload is more computationally demanding than the sharpen workload.

SGEMM and DGEMM

GEMM is "general matrix multiplication." Matrix multiplication is a fundamental mathematical operation. It is used in physical simulations, signal processing, graphics processing, and many other areas.

SFFT and DFFT

The fast Fourier transform (FFT) workloads simulate the frequency analysis used to compute the spectrum view in an audio processing application such as Pro Tools.

N-Body

This workload computes a physical simulation similar to that required for a physics game placed in outer space.

Ray Trace

The ray trace workload renders a 3D scene from a geometric description.

Geekbench 3 Multi Core - Integer Operations

CPU Integer test suite from Geekbench 3

AES Encryption

The AES workload encrypts a generated text string using the advanced encryption standard (AES). AES is used in security tools such as SSL, IPsec, and GPG. Geekbench uses the AES-NI instructions when they are available. When the AES-NI instructions are not available, Geekbench uses its own software AES implementation.

Twofish

The Twofish workload also encrypts a text string, but it uses the Twofish algorithm. Twofish is from the family of encryption algorithms known as "Feistel ciphers." It is included in the OpenPGP standard.

SHA1

SHA1 is a cryptographic hash algorithm: given a binary input it generates a "hash" or "digest" of the input. SHA1 is designed so that the hash may be computed quickly, but it is difficult to find a string that generates a given hash. SHA1 may be used, for example, to encrypt passwords by storing the hash instead of the password text. The SHA1 workload uses a text string as input.

SHA2

SHA2 solves the same problem as SHA1, but is more secure: SHA1 has a known vulnerability to "collision attacks." Although these attacks are still impractical and SHA1 is still widely used, it is being gradually replaced by SHA2.

BZip2 Compression and Decompression

BZip2 is a compression algorithm. The BZip2 workloads compress and decompress an ebook formatted using HTML. Geekbench 3 uses bzlib version 1.0.6 in the BZip2 workloads.

JPEG Compression and Decompression

The JPEG workloads compress and decompress one digital image using lossy JPEG format. The workloads use libjpeg version 6b.

PNG Compression and Decompression

The PNG workloads also compress and decompress a digital image, but they do so using the PNG format. The workloads use libpng 1.6.2.

Sobel

The "Sobel operator" is used in image processing for finding edges in images. The Sobel workload uses the same input image as the JPEG and PNG workloads.

Lua

Lua is lightweight scripting language. The Lua workload is similar to the code used to display Geekbench results in the Geekbench Browser.

Djikstra

The Dijkstra workload computes driving directions between a sequence of destinations. Similar techniques are used by AIs to compute paths in games and by network routers to route computer network traffic.

Geekbench 3 Multi Core - Floating Point Operations

CPU Floating point test suite from Geekbench 3

Black Scholes

The Black-Scholes equation is used to model option prices on financial markets. The Black-Scholes workload computes the Black-Scholes formula: a special case solution of the Black-Scholes equation for European call and put options.

Mandelbrot

The Mandelbrot set is a fractal. It is a useful floating point workload because it has a low memory bandwidth requirement.

Sharpen Image

The sharpen image workload uses a standard image sharpening technique similar to those found in Photoshop or Gimp.

Blur Image

Image blurring is also found in tools such as Photoshop. In Geekbench 3, the blur image workload is more computationally demanding than the sharpen workload.

SGEMM and DGEMM

GEMM is "general matrix multiplication." Matrix multiplication is a fundamental mathematical operation. It is used in physical simulations, signal processing, graphics processing, and many other areas.

SFFT and DFFT

The fast Fourier transform (FFT) workloads simulate the frequency analysis used to compute the spectrum view in an audio processing application such as Pro Tools.

N-Body

This workload computes a physical simulation similar to that required for a physics game placed in outer space.

Ray Trace

The ray trace workload renders a 3D scene from a geometric description.

RAMspeed SMP

A multi-processor memory and cache performance test

Geekbench 3 Single Core - Memory Performance

Memory test suite from Geekbench 3

STREAM Copy

The stream copy workload tests how fast your computer can copy large amounts of data in memory. It executes a value-by-value copy of a large list of floating point numbers.

STREAM Scale

This workload is similar to stream copy, but each value is multiplied by a constant during the copy.

STREAM Add

The stream add workload reads two large lists of floating point numbers value-by-value, adds corresponding values, and stores the result in a third list.

STREAM Triad

This workload combines stream add and stream scale. It reads two lists of floating point numbers value-by-value, multiplies one of the numbers by a constant, adds the result to the other number, and writes that result to a third list.

Geekbench 3 Multi Core - Memory Performance

Memory test suite from Geekbench 3

STREAM Copy

The stream copy workload tests how fast your computer can copy large amounts of data in memory. It executes a value-by-value copy of a large list of floating point numbers.

STREAM Scale

This workload is similar to stream copy, but each value is multiplied by a constant during the copy.

STREAM Add

The stream add workload reads two large lists of floating point numbers value-by-value, adds corresponding values, and stores the result in a third list.

STREAM Triad

This workload combines stream add and stream scale. It reads two lists of floating point numbers value-by-value, multiplies one of the numbers by a constant, adds the result to the other number, and writes that result to a third list.

Apachebench

This test profile measures how many requests per second a given Apache server can sustain when carrying out 700,000 requests with 100 requests being carried out concurrently.

PHPbench

The test benchmarks various aspects of the PHP interpreter. PHPBench can be used to compare hardware, operating systems, PHP versions, PHP accelerators and caches, compiler options, etc.

Geekbench 3 Single Core

A CPU and RAM benchmark that generates a comparable, overall score of the system.

Geekbench 3 Multi Core

A CPU and RAM benchmark that generates a comparable, overall score of the system.

Unixbench Single Core

The purpose of UnixBench is to provide a basic indicator of the performance of a Unix-like system; hence, multiple tests are used to test various aspects of the system's performance.

Dhrystone

Developed by Reinhold Weicker in 1984. This benchmark is used to measure and compare the performance of computers. The test focuses on string handling, as there are no floating point operations. It is heavily influenced by hardware and software design, compiler and linker options, code optimization, cache memory, wait states, and integer data types.

Whetstone

This test measures the speed and efficiency of floating-point operations. This test contains several modules that are meant to represent a mix of operations typically performed in scientific applications. A wide variety of C functions including sin, cos, sqrt, exp, and log are used as well as integer and floating-point math operations, array accesses, conditional branches, and procedure calls. This test measure both integer and floating-point arithmetic.

Execl Throughput

This test measures the number of execl calls that can be performed per second. Execl is part of the exec family of functions that replaces the current process image with a new process image. It and many other similar commands are front ends for the function execve().

File Copy

This measures the rate at which data can be transferred from one file to another, using various buffer sizes. The file read, write and copy tests capture the number of characters that can be written, read and copied in a specified time (default is 10 seconds).

Pipe Throughput

A pipe is the simplest form of communication between processes. Pipe throughtput is the number of times (per second) a process can write 512 bytes to a pipe and read them back. The pipe throughput test has no real counterpart in real-world programming.

Pipe-based Context Switching

This test measures the number of times two processes can exchange an increasing integer through a pipe. The pipe-based context switching test is more like a real-world application. The test program spawns a child process with which it carries on a bi-directional pipe conversation.

Process Creation

This test measure the number of times a process can fork and reap a child that immediately exits. Process creation refers to actually creating process control blocks and memory allocations for new processes, so this applies directly to memory bandwidth. Typically, this benchmark would be used to compare various implementations of operating system process creation calls.

Shell Scripts

The shells scripts test measures the number of times per minute a process can start and reap a set of one, two, four and eight concurrent copies of a shell scripts where the shell script applies a series of transofrmation to a data file.

System Call Overhead

This estimates the cost of entering and leaving the operating system kernel, i.e. the overhead for performing a system call. It consists of a simple program repeatedly calling the getpid (which returns the process id of the calling process) system call. The time to execute such calls is used to estimate the cost of entering and exiting the kernel.

Unixbench Multi Core

The purpose of UnixBench is to provide a basic indicator of the performance of a Unix-like system; hence, multiple tests are used to test various aspects of the system's performance.