Generates and manages code for Protocol Buffers with features like clean data class generation, JSON serialization, oneof handling as sealed classes, and support for proto2 and proto3 syntaxes.
Pbandk is a Kotlin Multiplatform code generator and runtime for Protocol Buffers.
NOTE: This is the documentation for the version of pbandk currently in development. Documentation for the latest stable version is available at https://github.com/streem/pbandk/blob/v0.16.0/README.md.
FieldMask)Read below for more information and see the examples.
This project is currently in beta. It has the core set of protobuf features implemented and is being used in production. But it is still under active development and new versions might introduce backwards-incompatible changes to support new features or to improve the library's usability in Kotlin. Pull requests are welcome for any of the "Not Yet Implemented" features above.
This project follows semantic versioning. After v1.0.0 is released, future versions will preserve backwards compatibility.
The project currently has a single maintainer (@garyp) working on it in his spare time. Contributors who would like to become additional maintainers are highly welcome. Your contributions don't have to be in the form of code and could also be documentation improvements, issue triage, community outreach, etc.
For support or discussion relating to pbandk, please use the GitHub Discussions on this project. You can also find some of us in the #pbandk channel of the Kotlin Slack instance. Please drop in and say hi!
For the following addressbook.proto file:
syntax = "proto3";
package tutorial;
import "google/protobuf/timestamp.proto";
message Person {
string name = 1;
int32 id = 2;
string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
string number = 1;
PhoneType type = 2;
}
repeated PhoneNumber phones = 4;
google.protobuf.Timestamp last_updated = 5;
}
message AddressBook {
repeated Person people = 1;
}
The following file will be generated at tutorial/addressbook.kt:
To see a full version of the file, see here. See the "Generated Code" section below under "Usage" for more details.
Pbandk's code generator leverages . Download the and make sure is on the . Then download the (if you're using a SNAPSHOT build of pbandk, you might want to instead download the ), rename it to , make the file executable (), and make sure it is on the . To generate code from and put the generated code in , run:
protoc --pbandk_out=src/main/kotlin sample.proto
The file is generated as sample.kt in the subdirectories specified by the package. Like other X_out arguments,
comma-separated options can be added to --pbandk_out before the colon and out dir path:
Multiple options can be added to a single --pbandk_out argument by separating them with commas.
In addition to running protoc manually, the
Protobuf Plugin for Gradle can be used. See
this example to see how.
The self-executing jar file doesn't work on Windows. Also protoc doesn't support finding
protoc-gen-pbandk.bat on the PATH. So it has to be specified explicitly as a plugin. Thus on
Windows you will first need to build protoc-gen-pbandk locally:
./gradlew :protoc-gen-pbandk:protoc-gen-pbandk-jvm:installDist
And then provide the full path to protoc:
protoc \
--pbandk_out=src/main/kotlin \
--plugin=protoc-gen-pbandk=/path/to/pbandk/protoc-gen-pbandk/jvm/build/install/protoc-gen-pbandk/bin/protoc-gen-pbandk.bat \
sample.proto
Pbandk's runtime library provides a Kotlin layer over the preferred Protobuf library for each platform. The libraries are present on Maven Central. Using Gradle:
repositories {
// This repository is only needed if using a SNAPSHOT version of pbandk
maven { url "https://s01.oss.sonatype.org/content/repositories/snapshots/" }
mavenCentral()
}
dependencies {
// Can be used from the `common` sourceset in a Kotlin Multiplatform project,
// or from platform-specific JVM, Android, JS, or Native sourcesets/projects.
implementation("pro.streem.pbandk:pbandk-runtime:0.16.1-SNAPSHOT")
}
Pbandk does not generate gRPC code itself, but offers a pbandk.gen.ServiceGenerator interface in
the protoc-gen-pbandk-lib-jvm project with a single method that can be implemented to generate the
code.
To do this, first depend on the project but it will only be needed at compile time because it's already there at runtime:
dependencies {
compileOnly("pro.streem.pbandk:protoc-gen-pbandk-lib:0.16.1-SNAPSHOT")
}
Then, the kotlin_service_gen option can be given to protoc to use the generator. The option is a path-separated
collection of JAR files to put on the classpath. It can end with a pipe (i.e. |) following by the fully-qualified
class name of the implementation of the ServiceGenerator to use. If the last part is not present, it will use the
ServiceLoader mechanism to find the first implementation to use. For example, to gen with from
, it might look like:
protoc --pbandk_out=kotlin_service_gen=gen.jar|my.Generator,kotlin_package=my.pkg:src/main/kotlin some.proto
For more details, see the custom-service-gen example.
The package is either the kotlin_package plugin option, the java_package protobuf option or the package set in the message. If the google.protobuf
package is referenced, it is assumed to be a well-known type and is changed to reference pbandk.wkt.
Each Protobuf message extends and has an method to encode the message with the Protobuf binary encoding into a . The companion object of every message has a method: it accepts a and returns an instance of the class. Each platform also provides additional and methods that are platform-specific. For example, the JVM provides and methods that operate on Java's and , respectively. There are also and methods that use the Protobuf JSON encoding to encode/decode the message into a string.
Messages are immutable Kotlin data classes. This means they automatically implement hashCode, equals, and
toString. Each class has an unknownFields map which contains information about extra fields the decoder didn't
recognize. If there are values in this map, they will be encoded on output. The MessageDecoder instances have a
constructor option to discard unknown fields when reading.
For proto3, the only nullable fields are messages, oneof fields, and optional fields. Other values have defaults. For
proto2, optional fields are nullable and defaulted as such. Types are basically the same as they would be in Java.
However, bytes fields actually use a pbandk.ByteArr class which is a simple wrapper around ByteArray. This was done
because Kotlin does not handle array fields in data classes predictably and it wasn't worth overriding and
every time.
Regardless of optimize_for options, the generated code is always the same. Each message has a protoSize field that
lazily calculates the size of the message when first invoked. Also, each message has the plus operator defined which
follows protobuf merge semantics.
Oneof fields are generated as nested classes of a common sealed base class. Each oneof inner field is a class that wraps a single value.
The parent message also contains a nullable field for every oneof inner field. This field resolves to the oneof inner field's value when the oneof is set to that inner field. Otherwise it resolves to null.
Enum fields are generated as sealed classes with a nested object for each known enum value, and a
Unrecognized nested class to hold unknown values. This is preferred over traditional Kotlin enum classes
because enums in protobuf are open ended and may not be one of the specific known values. Traditional
enum classes would not be able to capture this state, and using sealed classes this way requires the
user to do explicit checks for the Unrecognized value during exhaustive when clauses.
Each enum object contains a value field with the numeric value of that enum, and a name field
with the string value of that enum. Developers should use the fromValue and fromName methods
present on the companion object of the sealed class to map from a numeric or string value,
respectively, to the corresponding enum object.
The values field on the companion object of the sealed class contains a list of all known enum
values.
repeated fields are normal Lists. Developers should make no assumptions about which list implementation is used.
maps are represented by Kotlin Maps. In proto2, due to how map entries are serialized, both the key and the value
are considered nullable.
Well known types (i.e. those in the google/protobuf imports) are shipped with the runtime under the pbandk.wkt package.
Specialized support is provided to map the types defined in google/protobuf/wrappers.proto into Kotlin nullable primitives (e.g. String? for google.protobuf.StringValue, Int? for google.protobuf.Int32Value, etc.).
Services can be handled with a custom service generator. See the "Service Code Generation" section above and the custom-service-gen example.
The project is built with Gradle and has several sub projects. They are:
To generate the protoc-gen-pbandk distribution, run:
./gradlew :protoc-gen-pbandk:protoc-gen-pbandk-jvm:assembleDist
If you want to make changes to pbandk, and immediately test these changes in your separate project,
first install the generator locally:
./gradlew :protoc-gen-pbandk:protoc-gen-pbandk-jvm:installDist
This puts the files in the build/install folder. Then you need to tell protoc where to find this plugin file.
For example:
protoc \
--plugin=protoc-gen-pbandk=/path/to/pbandk/protoc-gen-pbandk/jvm/build/install/protoc-gen-pbandk/bin/protoc-gen-pbandk \
--pbandk_out=src/main/kotlin \
src/main/proto/*.proto
This will generate kotlin files for the specified *.proto files, without needing to publish first.
To build the runtime library for both JS and the JVM, run:
./gradlew :pbandk-runtime:assemble
If any changes are made to the generated code that is output by protoc-gen-pbandk, then the
well-known types (and other proto types used by pbandk) need to be re-generated using the updated
protoc-gen-pbandk binary:
./gradlew generateProtos
Important: If making changes in both the :protoc-gen-pbandk:protoc-gen-pbandk-lib and :pbandk-runtime projects at
the same time, then it's likely the :pbandk-runtime:generateWellKnownTypeProtos task will fail to compile. To work
around this, stash the changes in the project, run the task with only
the changes, and then unstash the changes and rerun the
task.
To run conformance tests, the conformance-test-runner must be built (does not work on Windows).
git clone -b v28.0 --depth 1 --recurse-submodules --shallow-submodules https://github.com/protocolbuffers/protobuf.git
cd protobuf
cmake -S . -B build -DCMAKE_CXX_STANDARD=14 -Dprotobuf_BUILD_CONFORMANCE=ON -Dprotobuf_BUILD_TESTS=OFF -Dprotobuf_BUILD_LIBUPB=OFF
cmake --build build --parallel=10
You should now have a conformance_test_runner available in protobuf/build directory. Test it by running ./conformance_test_runner --help
Set the CONF_TEST_PATH environment variable (used to run the tests below) with:
export CONF_TEST_PATH="$(pwd)/conformance_test_runner"
Now, back in pbandk, build all conformance sub-projects via:
./gradlew :conformance:conformance-lib:assemble \
:conformance:conformance-jvm:installDist \
:conformance:conformance-native:build
You are now ready to run the conformance tests. Make sure CONF_TEST_PATH environment variable is set to path/to/protobuf/build/conformance_test_runner (see above).
Then, from the root directory:
./conformance/test-conformance.sh
Note that by default, the test-conformance.sh script will run the conformance test for jvm, js, wasmJs, and linux. This will fail when running them on MacOS due to missing linux binaries. So in that case, run the tests for each platform individually:
./conformance/test-conformance.sh jvm
./conformance/test-conformance.sh js
./conformance/test-conformance.sh wasmJs
./conformance/test-conformance.sh macos
Releases are handled automatically via CI once the git tag is created.
Setup a couple shell variables to simplify the rest of the commands below:
export VERSION="0.9.0"
export NEXT_VERSION="0.9.1"
To create a new release:
Then prepare the repository for development of the next version:
GitHub will build and publish the new release once it sees the new tag:
git push origin --follow-tags master.gh release create "v${VERSION}" -F <(git tag -l --format='%(contents)' "v${VERSION}").This repository was originally forked from https://github.com/cretz/pb-and-k. Many thanks to https://github.com/cretz for creating this library and building the initial feature set.
pbandk uses its own pure-Kotlin protobuf implementation that is heavily based on the Google Protobuf Java library.
StringValue, BoolValue) as nullable primitives (String?, Boolean?, etc.)deprecated protobuf option when generating Kotlin codejs)
@JsExport functionality.wasmJs)
mingwX64)
mingwX64 platform. The official protobuf project does not yet support running the protobuf conformance tests on Windows, so we cannot fully validate pbandk's correctness when run on Windows. However, almost all of the mingwX64 pbandk code is shared with the other native platforms (Linux, macOS), so the passing conformance tests on those platforms still provide a pretty good guarantee of pbandk's correctness on mingwX64.(pbandk.PublicForGeneratedCode::class)
package tutorial
.Export
public data class Person(
val name: String = "",
val id: Int = 0,
val email: String = "",
val phones: List<tutorial.Person.PhoneNumber> = emptyList(),
val lastUpdated: pbandk.wkt.Timestamp? = null,
override val unknownFields: Map<Int, pbandk.UnknownField> = emptyMap()
) : pbandk.Message {
override operator fun plus(other: pbandk.Message?): tutorial.Person = protoMergeImpl(other)
override val descriptor: pbandk.MessageDescriptor<tutorial.Person> get() = Companion.descriptor
override val protoSize: Int by lazy { super.protoSize }
public companion object : pbandk.Message.Companion<tutorial.Person> {
public val defaultInstance: tutorial.Person by lazy { tutorial.Person() }
override fun decodeWith(u: pbandk.MessageDecoder): tutorial.Person = tutorial.Person.decodeWithImpl(u)
override val descriptors: pbandk.MessageDescriptor<tutorial.Person> = pbandk.MessageDescriptor(
fullName = "tutorial.Person",
messageClass = tutorial.Person::class,
messageCompanion = this,
fields = buildList(5) {
add(
pbandk.FieldDescriptor(
messageDescriptor = this@Companion::descriptor,
name = "name",
number = 1,
type = pbandk.FieldDescriptor.Type.Primitive.String(),
jsonName = "name",
value = tutorial.Person::name
)
)
add(
pbandk.FieldDescriptor(
messageDescriptor = this@Companion::descriptor,
name = "id",
number = 2,
type = pbandk.FieldDescriptor.Type.Primitive.Int32(),
jsonName = "id",
value = tutorial.Person::id
)
)
add(
pbandk.FieldDescriptor(
messageDescriptor = this@Companion::descriptor,
name = "email",
number = 3,
type = pbandk.FieldDescriptor.Type.Primitive.String(),
jsonName = "email",
value = tutorial.Person::email
)
)
add(
pbandk.FieldDescriptor(
messageDescriptor = this@Companion::descriptor,
name = "phones",
number = 4,
type = pbandk.FieldDescriptor.Type.Repeated<tutorial.Person.PhoneNumber>(valueType = pbandk.FieldDescriptor.Type.Message(messageCompanion = tutorial.Person.PhoneNumber.Companion)),
jsonName = "phones",
value = tutorial.Person::phones
)
)
add(
pbandk.FieldDescriptor(
messageDescriptor = this@Companion::descriptor,
name = "last_updated",
number = 5,
type = pbandk.FieldDescriptor.Type.Message(messageCompanion = pbandk.wkt.Timestamp.Companion),
jsonName = "lastUpdated",
value = tutorial.Person::lastUpdated
)
)
}
)
}
public sealed class PhoneType(override val value: Int, override val name: String? = null) : pbandk.Message.Enum {
override fun equals(other: kotlin.Any?): Boolean = other is tutorial.Person.PhoneType && other.value == value
override fun hashCode(): Int = value.hashCode()
override fun toString(): String = "Person.PhoneType.${name ?: "UNRECOGNIZED"}(value=$value)"
public object MOBILE : PhoneType(0, "MOBILE")
public object HOME : PhoneType(1, "HOME")
public object WORK : PhoneType(2, "WORK")
public class UNRECOGNIZED(value: Int) : PhoneType(value)
public companion object : pbandk.Message.Enum.Companion<tutorial.Person.PhoneType> {
public val values: List<tutorial.Person.PhoneType> by lazy { listOf(MOBILE, HOME, WORK) }
override fun fromValue(value: Int): tutorial.Person.PhoneType = values.firstOrNull { it.value == value } ?: UNRECOGNIZED(value)
override fun fromName(name: String): tutorial.Person.PhoneType = values.firstOrNull { it.name == name } ?: throw IllegalArgumentException("No PhoneType with name: $name")
}
}
public data class PhoneNumber(
val number: String = "",
val type: tutorial.Person.PhoneType = tutorial.Person.PhoneType.fromValue(0),
override val unknownFields: Map<Int, pbandk.UnknownField> = emptyMap()
) : pbandk.Message {
override operator fun plus(other: pbandk.Message?): tutorial.Person.PhoneNumber = protoMergeImpl(other)
override val descriptor: MessageDescriptor<tutorial.Person.PhoneNumber> get() = Companion.descriptor
override val protoSize: Int by lazy { super.protoSize }
public companion object : pbandk.Message.Companion<tutorial.Person.PhoneNumber> {
public val defaultInstance: tutorial.Person.PhoneNumber by lazy { tutorial.Person.PhoneNumber() }
override fun decodeWith(u: pbandk.MessageDecoder): tutorial.Person.PhoneNumber = tutorial.Person.PhoneNumber.decodeWithImpl(u)
override val descriptor: pbandk.MessageDescriptor<tutorial.Person.PhoneNumber> = pbandk.MessageDescriptor(
fullName = "tutorial.Person.PhoneNumber",
messageClass = tutorial.Person.PhoneNumber::class,
messageCompanion = this,
fields = buildList(2) {
add(
pbandk.FieldDescriptor(
messageDescriptor = this@Companion::descriptor,
name = "number",
number = 1,
type = pbandk.FieldDescriptor.Type.Primitive.String(),
jsonName = "number",
value = tutorial.Person.PhoneNumber::number
)
)
add(
pbandk.FieldDescriptor(
messageDescriptor = this@Companion::descriptor,
name = "type",
number = 2,
type = pbandk.FieldDescriptor.Type.Enum(enumCompanion = tutorial.Person.PhoneType.Companion),
jsonName = "type",
value = tutorial.Person.PhoneNumber::type
)
)
}
)
}
}
}
.Export
public data class AddressBook(
val people: List<tutorial.Person> = emptyList(),
override val unknownFields: Map<Int, pbandk.UnknownField> = emptyMap()
) : pbandk.Message {
override operator fun plus(other: pbandk.Message?): tutorial.AddressBook = protoMergeImpl(other)
override val descriptor: MessageDescriptor<tutorial.AddressBook> get() = Companion.descriptor
override val protoSize: Int by lazy { super.protoSize }
public companion object : pbandk.Message.Companion<tutorial.AddressBook> {
public val defaultInstance: tutorial.AddressBook by lazy { tutorial.AddressBook() }
override fun decodeWith(u: pbandk.MessageDecoder): tutorial.AddressBook = tutorial.AddressBook.decodeWithImpl(u)
override val descriptor: pbandk.MessageDescriptor<tutorial.AddressBook> = pbandk.MessageDescriptor(
fullName = "tutorial.AddressBook",
messageClass = tutorial.AddressBook::class,
messageCompanion = this,
fields = buildList(1) {
add(
pbandk.FieldDescriptor(
messageDescriptor = this@Companion::descriptor,
name = "people",
number = 1,
type = pbandk.FieldDescriptor.Type.Repeated<tutorial.Person>(valueType = pbandk.FieldDescriptor.Type.Message(messageCompanion = tutorial.Person.Companion)),
jsonName = "people",
value = tutorial.AddressBook::people
)
)
}
)
}
}
.Export
.JsName("orDefaultForPerson")
public fun Person?.orDefault(): tutorial.Person = this ?: Person.defaultInstance
.Export
.JsName("orDefaultForPersonPhoneNumber")
public fun Person.PhoneNumber?.orDefault(): tutorial.Person.PhoneNumber = this ?: Person.PhoneNumber.defaultInstance
.Export
.JsName("orDefaultForAddressBook")
public fun AddressBook?.orDefault(): tutorial.AddressBook = this ?: AddressBook.defaultInstance
// Omitted multiple supporting private extension methods
protocprotocPATHprotoc-gen-pbandkchmod +x protoc-gen-pbandkPATHsample.protosrc/main/kotlinTo explicitly set the Kotlin package to my.pkg, use the kotlin_package option like so:
protoc --pbandk_out=kotlin_package=my.pkg:src/main/kotlin sample.proto
If you have multiple proto packages, you can map them using kotlin_package_mapping option like so:
protoc --pbandk_out=kotlin_package_mapping="simple.package->new.package;foo.bar.*->my.foo.bar.*":src/main/kotlin sample.proto
By default all generated classes have public visibility. To change the visibility to internal, use the
visibility option like so:
protoc --pbandk_out=visibility=internal:src/main/kotlin sample.proto
To log debug logs during generation, log=debug can be set as well.
my.Generatorgen.jarpbandk.MessageencodeToByteArrayByteArraydecodeFromByteArrayByteArrayencodeTo*decodeFrom*encodeToStreamdecodeFromStreamOutputStreamInputStreamencodeToJsonStringdecodeFromJsonStringequalshashCodeconformance/js - Conformance test runner for Kotlin/JSconformance/jvm - Conformance test runner for Kotlin/JVMconformance/native - Conformance test runner for Kotlin/Nativeconformance/wasmJs - Conformance test runner for Kotlin/Wasmconformance/lib - Common multiplatform code for conformance testsprotoc-gen-pbandk/jvm - Kotlin/JVM implementation of the code generator (can generate code for any platform, but requires JVM to run)protoc-gen-pbandk/lib - Multiplatform code (only Kotlin/JVM supported at the moment) for the code generator and ServiceGenerator libraryprotos - Protobuf definitions of the Well-Known Types, packaged as a separate project for compatibility with the Protobuf Gradle Pluginruntime - Kotlin Multiplatform library for runtime Protobuf supporttest-types - Protobuf definitions and generated code for protobuf messages used in pbandk unit tests:pbandk-runtimegenerateWellKnownTypeProtos:protoc-gen-pbandk:protoc-gen-pbandk-lib:pbandk-runtimegenerateWellKnownTypeProtosCHANGELOG.md: add a date for the release version, and update the release version's GitHub compare link with a tag instead of HEAD.
CHANGELOG.mdgradle.properties, README.md, and examples/*/build.gradle.kts to remove the SNAPSHOT suffix. For example, if the current version is 0.9.0-SNAPSHOT, then update it to be 0.9.0.README.md and update it to point at the new version. Also update the link to protoc-gen-pbandk-jvm in README.md to point at the new version.git commit -m "Bump to ${VERSION}" -a.git tag -a -m "See https://github.com/streem/pbandk/blob/v${VERSION}/CHANGELOG.md" "v${VERSION}".CHANGELOG.md: add a section for NEXT_VERSION that will follow the released version (e.g. if releasing 0.9.0 then add a section for 0.9.1).
CHANGELOG.mdgradle.properties, README.md, and examples/*/build.gradle.kts to ${NEXT_VERSION}-SNAPSHOT. For example, 0.9.1-SNAPSHOT.README.md.git commit -m "Bump to ${NEXT_VERSION}-SNAPSHOT" -a.Surfaced from shared tags and platforms — no rankings paid for.