Dev Log #001 - LibCBOR and LibJSON

Hi! Welcome to my dev logs, a somewhat informal place where I gather my thoughts after doing some programming. This first one is dedicated to two small libraries I put together after dealing with various other ones. Note that I mostly write my code in C++, primarily because of personal preference. I find the language fits my style quite well, even though it's not great in many aspects.

The formats we'll be dealing with in this post are CBOR (RFC 8949 or website), and JSON (RFC 8259 or website). I chose CBOR as my binary format of choice for simple storage of various things. It's versatile, easy to both encode and decode, and it maps nicely onto most data. That's one of the nicer properties it inherited from JSON. The second is JSON, but mostly as a grudging admission that it's unfortunately unavoidable in anything web related these days¹.

My motivation for writing these libraries was two-fold. First, I wanted a nicer library API than what many established libraries provide. Second, I wanted to understand both formats properly. I don't like staring into blackboxes. JSON might seem trivial at a glance, but parsing it has some intricacies, even if it's woefully underspecified.

On the API side, I think libraries usually pan out one of two ways. Either the API gives the user a lot of tools and customization points. That is a great property, but it often makes the library difficult to handle for beginners. Wrong use often blows up in unexpected places, and it's easy to introduce various vulnerabilities just by incorrectly prodding into it. The other extreme is something like nlohmann/json. It's a great library, and it serves its stated purpose of treating JSON as close to a native type as possible. However, due to that decision, it introduces nontrivial overhead by materializing the DOM for even the simplest of tasks.

I wanted a middle ground. I want an API that is simple to use, hard to mess up, and reasonably fast without resorting to SIMD tricks like what simdjson uses. Don't get me wrong, that library is a marvel of engineering, and it's genuinely impressive, but it imposes requirements on its input buffers, and I don't find it particularly easy to use.

The CBOR situation is somewhat more complicated. It's a newer, and much less popular format. Some libraries exist (and nlohmann/json supports it to an extent), but the ecosystem definitely isn't as crystallized as for JSON, where the aforementioned libraries are the default choices.

CBOR

After looking at what libraries were available at the time for CBOR around a year ago, I wasn't satisfied (a list is here). I wanted an API where I could easily pull out (almost) arbitrary data directly out of an input buffer, ideally without introducing allocations in the Encoder/Decoder, and leaving the allocation choice to the user as much as possible. That led me down to what the API actually looks like now, you can see a small (untested) example below.

#include <CBOR/Encoder.hpp>
#include <CBOR/Decoder.hpp>
#include <CBOR/DecoderHooks.hpp>

#include <array>
#include <compare>
#include <cstdlib>

struct MyData
{
    std::uint32_t Id;
    std::string   Name;
    double        Speed;

    constexpr auto operator<=>(const MyData &) const = default;
};

void EncodeHook(CBOR::Encoder &enc, const MyData &value)
{
    enc.EncodeMap(
        "id",    value.Id,
        "name",  value.Name,
        "speed", value.Speed
    );
}

void DecodeHook(CBOR::Decoder &dec, MyData &value)
{
    constexpr CBOR::StructHelper helper {
        CBOR::StructMember { "id",    &MyData::Id    },
        CBOR::StructMember { "name",  &MyData::Name  },
        CBOR::StructMember { "speed", &MyData::Speed },
    };
    helper.Decode(dec.AsItem(), value);
}

int main()
{
    std::array<std::uint8_t, 1024> buffer {};

    MyData data {
        .Id    = 10,
        .Name  = "John Doe",
        .Speed = 5.0,
    };

    CBOR::Encoder enc(buffer);
    enc.Encode(data);

    CBOR::ConstBuffer encoded(buffer.data(), enc.Size());
    CBOR::Decoder dec(encoded);

    MyData decoded = dec.Decode<MyData>();

    return data == decoded ? EXIT_SUCCESS : EXIT_FAILURE;
}

This is all that's needed to round trip MyData to and from any CBOR buffer. All that's needed is for the Encoder/Decoder to have access to the necessary hooks at the callsite via ADL. There are several nice properties that fall out of this design:

The Encode and Decode sides can be asymmetrical, based on the usecase. Unused fields are either not encoded or skipped during decoding.
Some fields can be left out deliberately, and finalizers can be called directly inside the hooks in case plain processing is not enough.
The deserialized structure can be completely non-allocating (in simple cases). Decoding a std::string_view or std::span<const std::uint8_t> with a few other values is perfectly valid. Even if allocations are needed, they are completely in the hook's control.
The hooks compose, so MyData could be a member inside another struct or class, and it can pass through without issue.

The underlying Encoder is split into two layers. The BasicEncoder exposes an interface that basically emits headers and data based on how it's driven. It requires discipline, because it's easy to emit incorrect data by mistake. That is wrapped by the actual Encoder, which includes support for most standard library data structures out of the box (std::string_view, std::string, std::vector, and more), and supports extensions via hooks. This is mostly a convenience layer where it's difficult to make mistakes, but it's still possible to provide a hook that uses the BasicEncoder directly when needed.

The Decoder is a little different. It's mostly stateless by itself, and it uses auxiliary types to drive the decoding process in a pull-based fashion. For example, arrays are decoded by an ArrayDecoder that exposes a Key() and Value() call, where each returns a generic ItemDecoder wrapper. The hooks serve as thin wrappers over that API, and the StructHelper hides most of the unordered handling and skipping detail behind a nicer interface (but it's worth noting that in general it's O(n²) where n is the # of specified data members). I learned afterwards that this is a fairly popular approach in Rust, where it's called the typestate pattern. Unfortunately, C++ can't produce the same level of soundness as Rust, but that's mostly a non-issue with some basic caution.

A nice consequence of this is that the user always has access to the underlying primitives. The easy option works well enough, but the Decoder can be driven directly with lookahead to produce a DOM. I used this functionality to create a special Printer that serves to produce a JSON-esque output for debugging.

I'm not claiming that this design is the best possible solution, but I find it achieves a nice balance. The user has explicit control over what happens, while being shielded from the format details. It requires some boilerplate, but I think that is acceptable. In the future, I might look into augmenting this with C++26 reflection, but I'm not that tempted by it so far, since I think this isn't obnoxious enough to engineer more "convenience" layers on top.

JSON

Once I had the API, and needed to work with JSON, I had a hammer and saw a nail. Funnily enough, I needed it as part of test tooling for a different format library, but that's a topic for a future dev log. Anyway, the point is that I think this API shape is nice, and even if I end up needing two hook functions per format per data structure, it's worth it. Of course, there are some considerations, but I don't want to get bogged down in the details right now, since this post is already dragging on.

Overall, the simple API for the JSON Parser and Printer pair can be very similar. Internally, the Printer is even simpler, we don't need a BasicEncoder alternative for JSON, since we're not dealing with raw bytes, and we can leverage standard formatting capabilities. The harder part is writing a proper Parser with the same style as the Decoder. Interestingly, it turned out to be less of a problem than I initially thought it would be. I found that it was really helpful to write a small pull-based lexer that produces small non-owning tokens. It was really easy to just check if the tokens are valid inside the helper types (almost identical naming convention to LibCBOR).

One compromise I made (partly out of laziness) was to support only doubles in the initial versions. This is partly because the primary Number type in JavaScript is a double under the hood, and JSON is invariably a very JS-native format. The implementation is internally ready for other types as well, but those will be added on an as-needed basis, if ever.

Conclusion

I think these two formats are a really nice exercise. They are fairly small, and it's easy enough to get a reasonably conformant implementation with decent performance in a few kLoC (I think both libraries are under 5k lines so far). I'm planning on using this convention in future projects as well, even when I'm building DOMs, because APIs like this are nice both to implement and use.

If you're interested in checking these libraries out, you can find them on my Forgejo instance (LibCBOR, LibJSON). The READMEs have some quickstart information, but I'm planning on introducing some more detailed documentation on this website as well (LibCBOR and LibJSON).

Thanks for reading, and see you next time!

1: This post articulates what's wrong with JSON in much more detail than I would be able to manage. Personally, I consider the lack of trailing commas and comments as a serious defect as well, even if it was explicitly designed not to support either.