General notes on storing BuiltMarketData in MongoDB


#1

Our services are built on SpringBoot so interfacing with MongoDB is quite trivial. We had a use case where we needed to store calibrated market data for a given date but were having difficulties creating a MongoDB custom serializer because it’s an immutable JodaBean and it’s quite deeply nested. We got around it by storing BuiltMarketData as a JSON blob (string) in the DB. Typically our BuiltMarketData objects have a JSON string size of 5.7MB, compressing the data gave us a modest decrease of 5.2MB at the expense of performance. Deserializing the BuiltMarketData in a binary format (byte array) didn’t help since the resultant byte array had to be base64 encoded and stored as a string anyway.

Key definition:

@BeanDefinition
public class MarketDataKey implements Bean, Cloneable, Serializable {

    @PropertyDefinition
    private LocalDate valuationDate;

    @PropertyDefinition
    private String timeZone;

    @PropertyDefinition
    private Instance instance;

    public static MarketDataKey of(LocalDate valDate, String timeZone, Instance instance) {
        MarketDataKey key = new MarketDataKey();
        key.setInstance(instance);
        key.setValuationDate(valDate);
        key.setTimeZone(timeZone);
        return key;
    }

MongoDB Document Definition:

@BeanDefinition
@Document(value = "marketData")
public class BuiltMarketDataDomain implements Bean, Cloneable, Serializable {

    // This is BuiltMarketData stored as a SLOB (string)
    @PropertyDefinition
    private String marketData;

    @PropertyDefinition
    private String id;

    @Id
    @PropertyDefinition
    private MarketDataKey key;

    @JsonIgnore
    public BuiltMarketData getBuiltMarketData() {
        return JodaBeanSer.COMPACT.jsonReader().read(marketData, BuiltMarketData.class);
    }

    public void setBuiltMarketData(BuiltMarketData marketData) {
        setMarketData(JodaBeanSer.COMPACT.jsonWriter().write(marketData));
    }

    public static BuiltMarketDataDomain of(BuiltMarketData md, LocalDate valDate, String timeZone, Instance instance) {
        BuiltMarketDataDomain bmd = new BuiltMarketDataDomain();
        bmd.setKey(MarketDataKey.of(valDate, timeZone, instance));

        bmd.setBuiltMarketData(md);

        return bmd;
    }

Spring Boot Repository:

public interface BuiltMarketDataRepository extends MongoRepository<BuiltMarketDataDomain, MarketDataKey> {

	boolean existsByKey(LocalDate date, Instance instance);

	BuiltMarketDataDomain findByKey(LocalDate date, Instance instance);

}

#2

We also use JSON/XML/binary extracts from Joda-Beans internally.