Minimal schema-based binary format enabling compact, readable schemas with manual byte-size control, enums and sealed-like subtypes, fixed/variable arrays, plus serializer/deserializer code generation and JSON Schema validation.
Buffela (pronounced bah-FEH-lah) is an extensible, schema-based binary format.
First, you write the schema in good old YAML:
Uuid: String(36)
Timestamp: Double
Gender:
- FEMALE
- MALE
Phone:
countryCode: Unsigned(10)
number: String
User:
userId: Uuid
Anonymous: {}
Registered:
verified: Boolean
Viewer:
birthDate: String # We'll implement our own custom "Date" type in the Custom Primitives section
gender: Gender
phone: Phone?
Organizer:
roles: String[UByte]
userId: String
email: String
AuthTokenPayload:
issuedAt: Timestamp
user: User
AuthTokenSignature:
hmac256: Bytes(32)
Then you can import it in your favorite programming language - as long as it is Javascript, Typescript or Kotlin ;) - and write type safe serialization and deserialization code.
Buffela supports all the types you would expect (strings, booleans, numbers), along with enums, subtypes (similar to kotlin's sealed types) and arrays (in both constant and variable sized variants). You can also easily extend buffela with your own types!
Warning! This version is not backwards compatible with packets generated using older versions.
Buffela does away with forward and backward compatible schemata, in favor of schema readability and size efficiency. For the example we'll be building in this guide, the output will end up being more than 30% smaller than the equivalent protobuf! To achieve this size efficiency, buffela gives you more control over the format. You can manually specify the byte length of numbers (byte, short, integer, long), or even the bit length if you're so inclined. The same goes for arrays, you can make them fixed size or specify the byte/bit length of the item count. Additionally, you don't have to worry about field numbers and type safety is built in.
See the equivalent protobuf schema for comparison and decide for yourself which one you would prefer writing.
We currently support Javascript/Typescript and Kotlin Multiplatform.
There are two ways to implement a (de)serializer:
Reflection
Go through the schema and decide how to encode/decode each field at runtime. This can be slow, especially for typed languages but it means that the schema needs no preprocessing.
Code generation
Decide and write down the steps to encode/decode each field in a preprocessing step. Basically, compile the schema into code. Encoding or decoding at runtime is as fast as it can be.
Ideally, we would support both approaches for all supported languages, but ain't nobody got time for that. So here is what we currently support:
| Language |
|---|
Install the schema parser
npm i @buffela/parser
You want to serialize?
npm i @buffela/serializer
You want to deserialize?
npm i @buffela/deserializer
You're building a front-end?
Install the buffer browser polyfill.
Install typescript as a dev dependency
npm i -D typescript
Set up a simple tsconfig.json inside your project folder (don't worry it's for your editor, I won't have you compile anything)
"compilerOptions": {
"moduleResolution": "nodenext",
"skipLibCheck": true
}
You'll need to install some dependencies required by the generated code in your project:
gr.elaevents.buffela:serialization
gr.elaevents.buffela:deserialization
Serialization library on maven central
Deserialization library on maven central
You can skip installing either package if you're interested in only serializing or only deserializing.
Install the developer tools as a dev dependency
npm i -D @buffela/tools
Run the compiler
buffela-js YOUR_SCHEMA OUTPUT_DIR
This will generate a .json and a .ts file in the specified directory with the same name as your schema file
If you don't want type safety you can skip generating the .ts file by setting the types option to an empty string like this: --types=
You can run the kotlin compiler through npm
Don't know what npm is? Bless your innocent soul xD. Install nvm (https://github.com/nvm-sh/nvm) and then run
nvm install --lts. Now you should have node and npm with it.
npx @buffela/tools-kotlin compile YOUR_SCHEMA OUTPUT_DIR --package=YOUR_PACKAGE
The first time around it will ask you to download the package, press Enter to proceed.
This will create a .kt file in the specified directory with the same name as your buffela schema.
package YOUR_PACKAGE
import gr.elaevents.buffela.serialization.serialize
import gr.elaevents.buffela.deserialization.deserialize
fun main() {
val bytes = AuthTokenPayload(
issuedAt = System.currentTimeMillis().toDouble(),
user = User.Registered.Viewer(
userId = ,
verified = ,
gender = Gender.MALE,
birthDate = ,
phone = Phone(
countryCode = 30u,
number = ,
),
)
).serialize()
payload = AuthTokenPayload.deserialize(bytes)
(payload.user User.Registered) {
println()
}
}
We provide a JSON schema for in-editor validation of your buffela schemata.
Note that while your editor highlights all errors, the error messages are not very descriptive. If you need more detailed errors, you can run the compiler for your language of choice.
Since we expect you'll be writing schemata in a development environment, the schema is bundled with the developer tools that you probably already need.
After installing the tools, you can find the schema at node_modules/@buffela/tools-common/schemata/buffela-schema.json.
You can export the JSON schema through the command line tool:
npx @buffela/tools-kotlin schema
This will create a buffela-schema.json file in your current directory.
Please make sure to run the command again after every minor or major buffela update in order to get the latest features.
Install on JetBrains products (Please select the latest schema version)
For VS Code, install the RedHat YAML extension and associate the JSON schema
The top level types inside your buffela schema are called root types. Their name must (1) start with a capital letter and (2) only contain letters and numbers.
Going back to our example:
Gender:
[...]
User:
[...]
AuthTokenPayload:
[...]
AuthTokenSignature:
[...]
Gender, User, AuthTokenPayload and AuthTokenSignature are all root types.
Root types can be either object types or enumeration types.
Enumeration types represent single-choice types.
They are arrays that can only contain unique values.
Their values must (1) be in all uppercase and (2) only contain letters, numbers and underscores.
Here is an enumeration type from our example:
Gender:
- FEMALE
- MALE
Object types represent a collection of fields.
They are objects that contain key-values pairs.
The keys are the field names, and they must (1) start with a lowercase letter and (2) only contain letters and numbers.
The values are the field types.
Here is an object from our example:
AuthTokenPayload:
issuedAt: Timestamp
user: User
You can reference other enumeration and object types. For example:
gender: Gender
user: User
These are the supported fixed-size primitive types along with their mapping to the supported languages and their size in bits.
Here is an example:
verified: Boolean
These are the supported constant-sized primitive types along with their mapping to the supported languages and their size in bits.
N must always be <= 31.
Here is an example:
countryCode: Unsigned(10)
String is a sentinel type meaning that you can optionally specify a constant length if you know it beforehand in order to save one byte.
Both of these examples are valid:
email: String
userId: String(36)
These are the supported typed array primitive types along with their mapping to the supported languages.
Typed array types require a parameter that specifies the length.
You must either pass a constant length (e.g. 10) or, if your array needs to have a variable length, you must pass the numeric type that will be used to store the length. Said type will constrain the maximum length of your array. The following types are accepted: UByte, UShort, Int and Unsigned(N).
All of these examples are valid:
vector: FloatArray(10) # This array must have a length of exactly 10 floats
vector: FloatArray(UByte) # This array can have a length of up to 255 floats
vector: FloatArray(Unsigned(10)) # This array can have a length of up to 1023 floats
The Bytes type maps to Uint8Array in JS and ByteArray in Kotlin. Like a typed array, it also requires a parameter that specifies the length, and the same rules apply.
You can add a [LENGTH] suffix to any type to create an array of that type. You can also chain them to create N-dimensional arrays. The length can be either a constant or a length type just like the parameter of typed arrays.
All of these examples are valid:
roles: String[10] # This array must have a length of exactly 10 strings
roles: String[UByte] # This array can have a length of up to 255 strings
roles: String[Unsigned(10)] # This array can have a length of up to 1023 strings
Don't get confused by typed arrays, you can also make them higher-dimensional:
spaceTemperature: FloatArray(10)[10][10] # This represents a 10x10x10 cube of floats
You can make any type optional (nullable) by adding a ? suffix. For example:
phone: Phone?
In our example, we store dates (specifically the birth date of registered viewers) as strings. This is not very efficient and takes 10 bytes (=80 bits) to store. With a custom type we could wind that down to only 22 bits. Specifically, we could combine the month and year into a single 17-bit number using the following formula: year * 12 + (month - 1) and store the day in a 5-bit number, allowing us to represent any date between 0000-00-00 and 10922-08-31. As this type requires math operations, we cannot use a simple object type but we can extend buffela with our own custom primitive.
A custom type consists of a serializer and a deserializer. We will have to make use of the SerializerBuffer and DeserializerBuffer APIs.
There are also some read-only properties:
SerializerBuffer.length: How many bytes have been written to the buffer thus farDeserializeBuffer.position: How many bytes have been read from the buffer thus farFor JSDoc or Typescript we need to create a folder called primitives alongside our schema type definition file. Inside the primitives folder we will create a .ts file named after our new type (in this case 'Date.ts'), write the interface, and make it the default export:
export default interface Date {
year: number;
month: number;
day: number;
}
We now need to create and register a serializer for our type. We do this in our registerSerializer call:
registerSerializer(schema, {
Date: {
serialize(buffer, value) {
const yearMonth = value.year * 12 + (value.month - 1)
const day = value.day - 1
buffer.writeUnsigned(yearMonth, 17)
buffer.writeUnsigned(day, )
}
}
})
Same story for the deserializer in our registerDeserializer call:
registerDeserializer(schema, {
Date: {
deserialize(buffer) {
const yearMonth = buffer.readUnsigned(17)
const day = buffer.()
{
: .(yearMonth / ),
: yearMonth % + ,
: day +
}
}
}
})
In Kotlin, custom types are data classes. The custom classes must be defined in the same package as the generated code. We begin by defining the class:
data class Date(val year: Int, val month: Int, val day: Int)
Then we extend SerializerBuffer with a method called writeType (in this case writeDate):
import gr.elaevents.buffela.serialization.SerializerBuffer
fun SerializerBuffer.writeDate(date: Date) {
val yearMonth = date.year * 12 + (date.month - 1)
val day = date.day - 1
writeUnsigned(yearMonth.toUInt(), 17)
writeUnsigned(day.toUInt(), 5)
}
Similarly, extend DeserializerBuffer with readType (readDate):
fun DeserializerBuffer.readDate(): Date {
val yearMonth = readUnsigned(17).toInt()
val day = readUnsigned(5).toInt()
return Date(
year = yearMonth / 12,
month = yearMonth % 12 + 1,
day = day + 1
)
}
Now we can finally use our type, so simply change
birthDate: String
to
birthDate: Date
You may find yourself using a type with a specific parameter or array length over and over again in your schema. That is why aliases exist. Simply define a type alias in your schema to any primitive or root type:
Uuid: String(36)
Your could also use aliases for purely semantic reasons:
Timestamp: Double
Have in mind though that alias resolution is not recursive, or put more simply, you cannot use aliases inside an alias.
Sometimes you may need an object that can take multiple forms. In our example we create an authentication token for all users, registered or not, containing a unique user id. But our registered users will have additional fields, like a flag indicating whether their credentials are verified. Additionally, we have two types of users: viewers and organizers. Organizers register by email and viewers register by phone. To represent this hierarchy we can use subtypes.
User:
[...]
Anonymous: {}
Registered:
[...]
Viewer:
[...]
Organizer:
[...]
In our example, Anonymous and Registered are subtypes of User. A subtype inherits all fields from its parent type, and can have other subtypes of its own. In the compiled representation the structure is flattened, meaning that all fields live in the same level.
Anonymous is kind of special in that it doesn't define any fields nor does it have any subtypes. In this case the value can be {} (an empty object).
How you specify a subtype differs from language to language:
In JavaScript, the compiled type will have an additional special field named _type. You need to provide a leaf schema type:
const schema = parseSchema(...)
const user = {
_type: schema.User.Registered.Viewer,
[...]
}
To check if an object is a subtype of a type you can use the instanceOf() function:
if (schema.User.Registered.instanceOf(user)) {
// Do something
}
In Kotlin, subtypes are just nested classes. So to create a registered viewer you would do:
val user = User.Registered.Viewer(
[...]
)
To check if an object is a subtype of a type you can use the is operator:
if (user is User.Registered) {
// Do something
}
You may need to override the type of a field only for specific subtypes. To do this, simply define the field again inside the subtype with the modified type. For our example, let's say that organizers can choose any user id they want, while all other users get automatically generated UUIDs. You would write this as:
User:
userId: Uuid # Resolves to String(36)
[...]
Registered:
[...]
Organizer:
userId: String
With this mechanism you can safely override the length parameter or an array's i-th dimension length (e.g. String -> String(10), Int[10] -> Int[UByte]). The only rule is that all type overrides should map to the same native type in your language of choice.
In advanced scenarios, you could also override with a more specific custom primitive (e.g Date -> DateTime).
In extreme scenarios, and if you're exclusively using Javascript, you could even override with a different numeric type (e.g. Int -> Double) as these both map to the same 'number' type. This, conversely, would be invalid in Kotlin.
In some cases you may want to serialize multiple root types into a single buffer. In this case you must work directly with the SerializerBuffer and DeserializerBuffer classes.
If you're using buffela for client-server communication you may want to add a version header to all packets.
const buffer = new SerializerBuffer()
schema.Header.serialize({ version: 2 }, buffer)
schema.Body.serialize({ ... }, buffer)
const bytes = buffer.toBytes()
const buffer = new DeserializerBuffer(bytes)
const header = schema.Header.deserialize(buffer)
if (header.version !== 2)
throw new Error("Packet version not supported")
const body = schema.Body.deserialize(buffer)
val buffer = SerializerBuffer()
val header = Header(version = 2)
header.serialize(buffer)
val body = Body(...)
body.serialize(buffer)
val bytes = buffer.toBytes()
val buffer = DeserializerBuffer(bytes)
val header = Header.deserialize(buffer)
if (header.version != 2)
throw IllegalStateException("Packet version not supported")
val body = Body.deserialize(buffer)
Buffela uses bit packing internally to minimize the output size, but that means that the boundary between consecutive types inside the buffer may not be byte-aligned. To ensure that the next serialize()/deserialize() call begins at a byte boundary, you can call the clearBitBuffer() method on the SerializerBuffer/DeserializerBuffer. This is useful when trying to isolate the bytes belonging to a specific type in order to hash or sign them. Note that if you clear the bit buffer between two serialize() calls, then you must also clear it between the deserialize() calls and vice versa.
In our example, we want to serialize our payload, then read it back as bytes in order to calculate the HMAC-256 signature and then write the calculated signature into the same buffer.
const buffer = new SerializerBuffer()
// Write the payload into the buffer
schema.AuthTokenPayload.serialize({ ... }, buffer)
// Read the serialized payload and sign it
const payloadBytes = buffer.()
hmac256 = (payloadBytes)
buffer.()
schema..({ hmac256 }, buffer)
bytes = buffer.()
const buffer = new Deserializer(bytes)
// Deserialize the payload
const payload = schema..(buffer)
payloadBytes = bytes.(, buffer.)
buffer.()
signature = schema..(buffer)
(payloadBytes, signature.)
View the complete example
val buffer = SerializerBuffer()
// Write the payload into the buffer
val payload = AuthTokenPayload(...)
payload.serialize(buffer)
// Read the serialized payload and sign it
val payloadBytes = payload.toBytes()
val hmac256 = sign(payloadBytes)
// Align to byte
buffer.clearBitBuffer()
// Write the signature into the buffer
val signature = AuthTokenSignature(hmac256 = hmac256)
signature.serialize(buffer)
// Get the combined payload + signature as bytes
val bytes = buffer.toBytes()
val buffer = DeserializerBuffer(bytes)
// Deserialize the payload
val payload = AuthTokenPayload.deserialize(buffer)
// The deserializer has only read the payload, all bytes up to the current position must be the payload
val payloadBytes = bytes.sliceArray(0 until buffer.position)
// Align to byte
buffer.clearBitBuffer()
// Continue to deserialize the signature
val signature = AuthTokenSignature.deserialize(buffer)
// Verify the signature
verify(payloadBytes, signature.hmac256)
View the complete example
| Reflection based serialization |
|---|
| Reflection based deserialization |
|---|
| Serializer code generation |
|---|
| Deserializer code generation |
|---|
| Javascript/Typescript | ✅ | ✅ | ❌ | ❌ |
| Kotlin Multiplatform | ❌ | ❌ | ✅ | ✅ |
const { parseSchema } = require('@buffela/parser')
const { registerSerializer } = require('@buffela/serializer')
const { registerDeserializer } = require('@buffela/deserializer')
/**
* Do this only if you have generated types
* @type {import('./YOUR_TYPES').default}
*/
const schema = parseSchema(require('./YOUR_JSON'))
registerSerializer(schema, {})
registerDeserializer(schema, {})
const bytes = schema.AuthTokenPayload.serialize({
issuedAt: Date.now(),
user: {
_type: schema.User.Registered.Viewer,
userId: '588809b0-d8ce-4a6b-a2aa-9b10fd9d7a11',
verified: true,
birthDate: '2003-07-22',
gender: schema.Gender.MALE,
phone: {
countryCode: 30,
number: '1234567890',
}
}
})
const payload = schema.AuthTokenPayload.deserialize(bytes)
if (schema.User.Registered.instanceOf(payload.user)) {
console.log('User is registered')
}
import { parseSchema } from '@buffela/parser'
import { registerSerializer } from '@buffela/serializer'
import { registerDeserializer } from '@buffela/deserializer'
import type Schema from './YOUR_TYPES'
import schemaObject from './YOUR_JSON'
const schema = parseSchema(schemaObject) as Schema
registerSerializer(schema, {})
registerDeserializer(schema, {})
const bytes = schema.AuthTokenPayload.serialize({
issuedAt: Date.now(),
user: {
_type: schema.User.Registered.Viewer,
userId: '588809b0-d8ce-4a6b-a2aa-9b10fd9d7a11',
verified: true,
birthDate: '2003-07-22',
gender: schema.Gender.MALE,
phone: {
countryCode: 30,
number: '1234567890',
}
}
})
const payload = schema.AuthTokenPayload.deserialize(bytes)
if (schema.User.Registered.instanceOf(payload.user)) {
console.log('User is registered')
}
| Buffela Type | Javascript Type | Kotlin Type | Bit Length | Description |
|---|
| Boolean | boolean | Boolean | 1 | True or False |
| Byte | number | Byte | 8 | Integers from -128 to 127 |
| UByte | number | UByte | 8 | Integers from 0 to 255 |
| Short | number | Short | 16 | Integers from -32,768 to 32,767 |
| UShort | number | UShort | 16 | Integers from 0 to 65,535 |
| Int | number | Int | 32 | Integers from -2,147,483,648 to 2,147,483,647 |
| UInt | number | UInt | 32 | Integers from 0 to 4,294,967,295 |
| Long | BigInt | Long | 64 | Integers from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 |
| ULong | BigInt | ULong | 64 | Integers from 0 to 18,446,744,073,709,551,615 |
| Float | number | Float | 32 | Decimals from 3.4 E-38 to 3.4 E +38 |
| Double | number | Double | 64 | Decimals from 1.7 E -308 to 1.7 E +308 |
| Buffela Type | Javascript Type | Kotlin Type | Bit Length | Description |
|---|
| Signed(N) | number | Int | N | Integers from -2^(N-1) to 2^(N-1) - 1 |
| Unsigned(N) | number | UInt | N | Integers from 0 to 2^N - 1 |
| Buffela Type | Javascript Type | Kotlin Type |
|---|
| UByteArray | Uint8Array | UByteArray |
| ByteArray | Int8Array | ByteArray |
| UShortArray | Uint16Array | UShortArray |
| ShortArray | Int16Array | ShortArray |
| UIntArray | Uint32Array | UIntArray |
| IntArray | Int32Array | IntArray |
| LongArray | BigInt64Array | LongArray |
| ULongArray | BigUint64Array | ULongArray |
| FloatArray | Float32Array | FloatArray |
| DoubleArray | Float64Array | DoubleArray |
| BooleanArray | boolean[] | BooleanArray |
| SerializerBuffer Methods | DeserializerBuffer Methods |
|---|
| writeByte(byte) | readByte() |
| writeUByte(uByte) | readUByte() |
| writeShort(short) | readShort() |
| writeUShort(uShort) | readUShort() |
| writeInt(int) | readInt() |
| writeUInt(uInt) | readUInt() |
| writeLong(long) | readLong() |
| writeULong(uLong) | readULong() |
| writeFloat(float) | readFloat() |
| writeDouble(double) | readDouble() |
| writeBoolean(boolean) | readBoolean() |
| writeString(string, nullTerminated = false) | readString(length = untilNull) |
| writeBytes(bytes) | readBytes(length) |
| writeSigned(value, bitLength) | readSigned(bitLength) |
| writeUnsigned(value, bitLength) | readUnsigned(bitLength) |
Surfaced from shared tags and platforms — no rankings paid for.