Line-by-line transliteration port of a mature Encoding Standard implementation. Streams and buffers convert between legacy encodings, UTF-8 and UTF-16, with mem utilities, label resolution and optional SIMD acceleration.
This is a Kotlin Multiplatform line-by-line transliteration port of hsivonen/encoding_rs.
Original Project: This port is based on hsivonen/encoding_rs. All design credit and project intent belong to the upstream authors; this repository is a faithful port to Kotlin Multiplatform with no behavioural changes intended.
This is an in-progress port. The goal is feature parity with the upstream Rust crate while providing a native Kotlin Multiplatform API. Every Kotlin file carries a // port-lint: source <path> header naming its upstream Rust counterpart so the AST-distance tool can track provenance.
hsivonen/encoding_rsThe text below is reproduced and lightly edited from
https://github.com/hsivonen/encoding_rs. It is the upstream project's own description and remains under the upstream authors' authorship; links have been rewritten to absolute upstream URLs so they continue to resolve from this repository.
encoding_rs an implementation of the (non-JavaScript parts of) the Encoding Standard written in Rust.
The Encoding Standard defines the Web-compatible set of character encodings, which means this crate can be used to decode Web content. encoding_rs is used in Gecko starting with Firefox 56. Due to the notable overlap between the legacy encodings on the Web and the legacy encodings used on Windows, this crate may be of use for non-Web-related situations as well; see below for links to adjacent crates.
Additionally, the mem module provides various operations for dealing with
in-RAM text (as opposed to data that's coming from or going to an IO boundary).
The mem module is a module instead of a separate crate due to internal
implementation detail efficiencies.
Due to the Gecko use case, encoding_rs supports decoding to and encoding from UTF-16 in addition to supporting the usual Rust use case of decoding to and encoding from UTF-8. Additionally, the API has been designed to be FFI-friendly to accommodate the C++ side of Gecko.
Specifically, encoding_rs does the following:
Additionally, encoding_rs::mem does the following:
std::ioNotably, the above feature list doesn't include the capability to wrap
a std::io::Read, decode it into UTF-8 and presenting the result via
std::io::Read. The encoding_rs_io
crate provides that capability.
no_std EnvironmentThe crate works in a no_std environment. By default, the alloc feature,
which assumes that an allocator is present is enabled. For a no-allocator
environment, the default features (i.e. alloc) can be turned off. This
makes the part of the API that returns Vec/String/Cow unavailable.
For decoding character encodings that occur in email, use the
charset crate instead of using this
one directly. (It wraps this crate and adds UTF-7 decoding.)
For mappings to and from Windows code page identifiers, use the
codepage crate.
This crate does not support single-byte DOS encodings that aren't required by
the Web Platform, but the oem_cp crate does.
Normalizing text into Unicode Normalization Form C prior to encoding text into
a legacy encoding minimizes unmappable characters. Text can be normalized to
Unicode Normalization Form C using the
icu_normalizer crate.
The exception is windows-1258, which after normalizing to Unicode Normalization
Form C requires tone marks to be decomposed in order to minimize unmappable
characters. Vietnamese tone marks can be decomposed using the
detone crate.
TL;DR: (Apache-2.0 OR MIT) AND BSD-3-Clause for the code and data combination.
Please see the file named COPYRIGHT.
The non-test code that isn't generated from the WHATWG data in this crate is under Apache-2.0 OR MIT. Test code is under CC0.
This crate contains code/data generated from WHATWG-supplied data. The WHATWG upstream changed its license for portions of specs incorporated into source code from CC0 to BSD-3-Clause between the initial release of this crate and the present version of this crate. The in-source licensing legends have been updated for the parts of the generated code that have changed since the upstream license change.
Generated API documentation is available online.
There is a long-form write-up about the design and internals of the crate.
An FFI layer for encoding_rs is available as a separate crate. The crate comes with a demo C++ wrapper using the C++ standard library and GSL types.
The bindings for the mem module are in the
encoding_c_mem crate.
For the Gecko context, there's a C++ wrapper using the MFBT/XPCOM types.
There's a write-up about the C++ wrappers.
There are currently these optional cargo features:
simd-accelEnables SIMD acceleration using the nightly-dependent portable_simd standard
library feature.
This is an opt-in feature, because enabling this feature opts out of Rust's guarantees of future compilers compiling old code (aka. "stability story").
Currently, this has not been tested to be an improvement except for these
targets and enabling the simd-accel feature is expected to break the build
on other targets:
If you use nightly Rust, you use targets whose first component is one of the above, and you are prepared to have to revise your configuration when updating Rust, you should enable this feature. Otherwise, please do not enable this feature.
Used by Firefox.
serdeEnables support for serializing and deserializing &'static Encoding-typed
struct fields using Serde.
Not used by Firefox.
fast-legacy-encodeA catch-all option for enabling the fastest legacy encode options. Does not affect decode speed or UTF-8 encode speed.
At present, this option is equivalent to enabling the following options:
fast-hangul-encodefast-hanja-encodefast-kanji-encodefast-gb-hanzi-encodefast-big5-hanzi-encodeAdds 176 KB to the binary size.
Not used by Firefox.
fast-hangul-encodeChanges encoding precomposed Hangul syllables into EUC-KR from binary search over the decode-optimized tables to lookup by index making Korean plain-text encode about 4 times as fast as without this option.
Adds 20 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
fast-hanja-encodeChanges encoding of Hanja into EUC-KR from linear search over the decode-optimized table to lookup by index. Since Hanja is practically absent in modern Korean text, this option doesn't affect perfomance in the common case and mainly makes sense if you want to make your application resilient agaist denial of service by someone intentionally feeding it a lot of Hanja to encode into EUC-KR.
Adds 40 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
fast-kanji-encodeChanges encoding of Kanji into Shift_JIS, EUC-JP and ISO-2022-JP from linear
search over the decode-optimized tables to lookup by index making Japanese
plain-text encode to legacy encodings 30 to 50 times as fast as without this
option (about 2 times as fast as with less-slow-kanji-encode).
Takes precedence over less-slow-kanji-encode.
Adds 36 KB to the binary size (24 KB compared to less-slow-kanji-encode).
Does not affect decode speed.
Not used by Firefox.
less-slow-kanji-encodeMakes JIS X 0208 Level 1 Kanji (the most common Kanji in Shift_JIS, EUC-JP and ISO-2022-JP) encode less slow (binary search instead of linear search) making Japanese plain-text encode to legacy encodings 14 to 23 times as fast as without this option.
Adds 12 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
fast-gb-hanzi-encodeChanges encoding of Hanzi in the CJK Unified Ideographs block into GBK and
gb18030 from linear search over a part the decode-optimized tables followed
by a binary search over another part of the decode-optimized tables to lookup
by index making Simplified Chinese plain-text encode to the legacy encodings
100 to 110 times as fast as without this option (about 2.5 times as fast as
with less-slow-gb-hanzi-encode).
Takes precedence over less-slow-gb-hanzi-encode.
Adds 36 KB to the binary size (24 KB compared to less-slow-gb-hanzi-encode).
Does not affect decode speed.
Not used by Firefox.
less-slow-gb-hanzi-encodeMakes GB2312 Level 1 Hanzi (the most common Hanzi in gb18030 and GBK) encode less slow (binary search instead of linear search) making Simplified Chinese plain-text encode to the legacy encodings about 40 times as fast as without this option.
Adds 12 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
fast-big5-hanzi-encodeChanges encoding of Hanzi in the CJK Unified Ideographs block into Big5 from
linear search over a part the decode-optimized tables to lookup by index
making Traditional Chinese plain-text encode to Big5 105 to 125 times as fast
as without this option (about 3 times as fast as with
less-slow-big5-hanzi-encode).
Takes precedence over less-slow-big5-hanzi-encode.
Adds 40 KB to the binary size (20 KB compared to less-slow-big5-hanzi-encode).
Does not affect decode speed.
Not used by Firefox.
less-slow-big5-hanzi-encodeMakes Big5 Level 1 Hanzi (the most common Hanzi in Big5) encode less slow (binary search instead of linear search) making Traditional Chinese plain-text encode to Big5 about 36 times as fast as without this option.
Adds 20 KB to the binary size.
Does not affect decode speed.
Not used by Firefox.
For decoding to UTF-16, the goal is to perform at least as well as Gecko's old uconv. For decoding to UTF-8, the goal is to perform at least as well as rust-encoding. These goals have been achieved.
Encoding to UTF-8 should be fast. (UTF-8 to UTF-8 encode should be equivalent
to memcpy and UTF-16 to UTF-8 should be fast.)
Speed is a non-goal when encoding to legacy encodings. By default, encoding to legacy encodings should not be optimized for speed at the expense of code size as long as form submission and URL parsing in Gecko don't become noticeably too slow in real-world use.
In the interest of binary size, by default, encoding_rs does not have encode-specific data tables beyond 32 bits of encode-specific data for each single-byte encoding. Therefore, encoders search the decode-optimized data tables. This is a linear search in most cases. As a result, by default, encode to legacy encodings varies from slow to extremely slow relative to other libraries. Still, with realistic work loads, this seemed fast enough not to be user-visibly slow on Raspberry Pi 3 (which stood in for a phone for testing) in the Web-exposed encoder use cases.
See the cargo features above for optionally making CJK legacy encode fast.
A framework for measuring performance is available separately.
It is a goal to support the latest stable Rust, the latest nightly Rust and the version of Rust that's used for Firefox Nightly.
At this time, there is no firm commitment to support a version older than
what's required by Firefox, and there is no commitment to treat MSRV changes
as semver-breaking, because this crate depends on cfg-if, which doesn't
appear to treat MSRV changes as semver-breaking, so it would be useless for
this crate to treat MSRV changes as semver-breaking.
As of 2024-11-01, MSRV appears to be Rust 1.40.0 for using the crate and
1.42.0 for doc tests to pass without errors about the global allocator.
With the simd-accel feature, the MSRV is even higher.
A compatibility layer that implements the rust-encoding API on top of encoding_rs is provided as a separate crate (cannot be uploaded to crates.io). The compatibility layer was originally written with the assuption that Firefox would need it, but it is not currently used in Firefox.
To regenerate the generated code:
portable_simd nightly feature of the standard library instead of the packed_simd crate. Only affects the simd-accel optional nightly feature.unsafe.rust-version to .packed_simd instead of packed_simd_2 again now that updates are back under the packed_simd name. Only affects the simd-accel optional nightly feature.build.rs. (This removal should resolve false positives reported by some antivirus products. This may break some build configurations that have opted out of Rust's guarantees against future build breakage.)no_std support.no_std environment (with alloc).simd-accel feature.packed_simd dependency to packed_simd_2.cfg-if dependency to 1.0.cfg-if updated to edition 2018 without a semver break.Decoder::latin1_byte_compatible_up_to return None in more
cases to make the method actually useful. While this could be argued
to be a breaking change due to the bug fix changing semantics, it does
not break callers that had to handle the None case in a reasonable
way anyway.convert_str_to_utf16.mem::convert_utf8_to_utf16_without_replacement.mem::utf8_latin1_up_to and mem::str_latin1_up_to.Decoder::latin1_byte_compatible_up_to.bincode (dev dependency) version requirement to 1.0.simd crate to packed_simd.simd-accel (README-only release).clippy:: prefix from clippy lint names.--features simd-accel work with stable-channel compiler to
simplify the Firefox build system.is_foo_bidi() not treat U+FEFF (ZERO WIDTH NO-BREAK SPACE
aka. BYTE ORDER MARK) as right-to-left.is_foo_bidi() functions report true if the input contains
Hebrew presentations forms (which are right-to-left but not in a
right-to-left-roadmapped block).convert_utf16_to_latin1_lossy.mem module assert that the input is in the range
U+0000...U+00FF (inclusive).mem module provide conversions from Latin1 and UTF-16 to UTF-8
that can deal with insufficient output space. The idea is to use them
first with an allocation rounded up to jemalloc bucket size and do the
worst-case allocation only if the jemalloc rounding up was insufficient
as the first guess.simd-accel-specific memory corruption introduced in
version 0.8.1 in conversions between UTF-16 and Latin1 in the mem module.#[inline(never)] annotation that was not meant for release.mem module to increase the performance when
converting long buffers.mem module.mem
module.Encoder::has_pending_state() public.simd crate dependency to 0.2.0.7F correctly in ISO-2022-JP.Hash for Encoding.InputEmpty correct precedence over OutputFull when encoding
with replacement and the output buffer passed in is too short or the
remaining space in the output buffer is too small after a replacement.PartialEq and Eq for the CoderResult, DecoderResult
and EncoderResult types.Encoder::encode_from_utf16. (Due to an oversight, it lacked the fix that
Encoder::encode_from_utf8 already had.)#[must_use].parallel-utf8).simd-accel is used.Encoder.encode_from_utf8_to_vec_without_replacement().Add Encoding.is_ascii_compatible().
Add Encoding::for_bom().
Make == for Encoding use name comparison instead of pointer comparison,
because uses of the encoding constants in different crates result in
different addresses and the constant cannot be turned into statics without
breaking other things.
The initial release.
dependencies {
implementation("io.github.kotlinmania:encoding-rs-kotlin:0.1.1")
}
./gradlew build
./gradlew test
See AGENTS.md and CLAUDE.md for translator discipline, port-lint header convention, and Rust → Kotlin idiom mapping.
This Kotlin port is distributed under the same Apache-2.0 license as the upstream hsivonen/encoding_rs. See LICENSE (and any sibling LICENSE-* / NOTICE files mirrored from upstream) for the full text.
Original work copyrighted by the encoding_rs authors.
Kotlin port: Copyright (c) 2026 Sydney Renee and The Solace Project.
Thanks to the hsivonen/encoding_rs maintainers and contributors for the original Rust implementation. This port reproduces their work in Kotlin Multiplatform; bug reports about upstream design or behavior should go to the upstream repository.
u16 / char16_t).u16 / char16_t) into a sequence of bytes in an Encoding
Standard-defined character encoding as if the lone surrogates had been
replaced with the REPLACEMENT CHARACTER before performing the encode.
(Gecko's UTF-16 is potentially invalid.)document.characterSet.https://github.com/hsivonen/encoding_c
next to the encoding_rs directory.https://github.com/hsivonen/codepage
next to the encoding_rs directory.https://github.com/whatwg/encoding
next to the encoding_rs directory.1d519bf8e5555cef64cf3a712485f41cd1a6a990 of the encoding repo.
(Note: f381389 was the revision of encoding used from before the encoding repo
license change.)encoding_rs directory as the working directory, run
python generate-encoding-data.py.usize instead of u8 at a time).alloc (with lesser API surface).std::simd unsafe slice access by larger types than u8/u16 to align_to.Cargo.tomlstatic when defining another static).is_single_byte() on Encoding.mem::decode_latin1() and mem::encode_latin1_lossy().replacementEncoding::for_name(). (Encoding::for_label(foo).unwrap() is
now close enough after the above label change.)parallel-utf8 cargo feature.&'static Encoding.Encoding from const to static
to make the referents unique across crates that use the refernces.FOO_INIT instances of Encoding to allow
foreign crates to initialize static arrays with references to Encoding
instances even under Rust's constraints that prohibit the initialization of
&'static Encoding-typed array items with &'static Encoding-typed
statics.const
to work so that cross-crate usage keeps the referents unique.Cows from Rust-only non-streaming methods for encode and decode.Encoding::for_bom() returns the length of the BOM.simd-accel feature flag. (Requires
nightly Rust.)Surfaced from shared tags and platforms — no rankings paid for.