kotlinx-benchmark

0.4.17indexed

Toolkit for running benchmarks on multiplatform code, offering low noise, reliable results, statistical analysis, and detailed performance reports. Supports various target configurations and custom benchmark profiles.

JSJVMNativeWasm·Kotlin/kotlinx-benchmark

635

Stars

—

Used by

dependents

Health

/ 100

kotlinx-benchmark

kotlinx-benchmark is a toolkit for running benchmarks for multiplatform code written in Kotlin. It is designed to work with Kotlin/JVM, Kotlin/JS, Kotlin/Native, and Kotlin/WasmJs and WasmWasi (experimental) targets.

To get started, ensure you're using Kotlin 2.2.0 or newer and the latest stable Gradle 8 release or newer. However, because the Kotlin/WasmJs target is experimental and in an active development phase, it guarantees support only for the specific Kotlin version used to build the library. For the latest version, this is Kotlin 2.2.0.

Features

Low noise and reliable results
Statistical analysis
Detailed performance reports

Useful guides

Setting Up a Kotlin Multiplatform Project for Benchmarking

To configure a Kotlin Multiplatform project for benchmarking, follow the steps below. If you want to benchmark only Kotlin/JVM and Java code, you may refer to our comprehensive guide dedicated to setting up benchmarking in those specific project types.

Target-specific configurations

To run benchmarks on a platform ensure your Kotlin Multiplatform project targets that platform. For different platforms, there may be distinct requirements and settings that need to be configured. The guide below contains the steps needed to configure each supported platform for benchmarking.

Kotlin/JVM

To run benchmarks in Kotlin/JVM:

Kotlin/JS

To run benchmarks in Kotlin/JS:

Create a JS target with Node.js execution environment:

// build.gradle.kts
kotlin {
    js { 
        nodejs() 
    }
}

// build.gradle.kts
benchmark {
    targets {
        register("js")
    }
}

Kotlin/Native

To run benchmarks in Kotlin/Native:

Create a Native target:

// build.gradle.kts
kotlin {
    linuxX64()
}

// build.gradle.kts
benchmark {
    targets {
        register("linuxX64")
    }
}

It is possible to register multiple native targets. However, benchmarks can be executed only for the host target. This library supports all targets supported by the Kotlin/Native compiler.

Kotlin/Wasm

To run benchmarks in Kotlin/Wasm:

Create a Wasm target with Node.js execution environment:

// build.gradle.kts
kotlin {
    wasmJs { 
        nodejs() 
    }
}

// build.gradle.kts
benchmark {
    targets {
        register("wasmJs")
    }
}

The similar configuration could be applied to compile WasmWasi target.

Note: Kotlin/Wasm is an experimental compilation target for Kotlin. It may be dropped or changed at any time. Refer to Kotlin/Wasm documentation for up-to-date information about the target stability.

Writing Benchmarks

After setting up your project and configuring targets, you can start writing benchmarks. As an example, let's write a simplified benchmark that tests how fast we can add up numbers in an ArrayList:

Your final benchmark class will look something like this:

Note: Benchmark classes located in the common source set will be run in all platforms, while those located in a platform-specific source set will be run only in the corresponding platform.

See writing benchmarks for a complete guide for writing benchmarks.

Running Benchmarks

To run your benchmarks in all registered platforms, run benchmark Gradle task in your project. To run only on a specific platform, run <target-name>Benchmark, e.g., jvmBenchmark.

For more details about the tasks created by the kotlinx-benchmark plugin, refer to this guide.

Benchmark Configuration Profiles

The kotlinx-benchmark library provides the ability to create multiple configuration profiles. The main configuration is already created by the toolkit. Additional profiles can be created as needed in the configurations section of the benchmark block:

// build.gradle.kts
benchmark {
    configurations {
        named("main") {
            warmups = 20
            iterations = 10
            iterationTime = 3
            iterationTimeUnit = "s"
        }
        register("smoke") {
            include("<pattern of fully qualified name>")
            warmups = 5
            iterations = 3
            iterationTime = 500
            iterationTimeUnit = "ms"
        }
    }
}

Refer to our comprehensive guide to learn about configuration options and how they affect benchmark execution.

Separate source set for benchmarks

Often you want to have benchmarks in the same project, but separated from main code, much like tests. Refer to our detailed documentation on configuring your project to set up a separate source set for benchmarks.

Analyzing results

Upon finishing benchmarks execution, the plugin prints out a summary table with results. In addition to that table, it also saves results into a file. By default, that would be a JSON file with a format compatible with the format used by JMH. The format is configured using reportFormat option, you can read more about it in the configuration documentation.

While runs consisting of a few benchmarks could be analyzed without any additional tools, analyzing run results consisting of dozens of benchmarks could quickly become tedious.

To facilitate this process, you can use Kotlin Notebooks, which provides means for manipulating, analyzing and visualizing data.

We prepared several notebooks demonstrating different analysis techniques using our example projects. Namely,

You can always copy these notebooks into your projects and adapt for your particular needs.

Examples

To help you better understand how to use the kotlinx-benchmark library, we've provided an examples subproject. These examples showcase various use cases and offer practical insights into the library's functionality.