Hi team, I want to confirm the bitmap ability on saving space for nested column (JSON). Say if we have a field called

attributes

, and it is a

JSON

type. The JSON value has three keys have an identical long string, for instance

{
  "key1": "same long string",
  "key2": "same long string",
  "key3": "same long string"
}

Would Druid only store the

same long string

once instead of thrice in the data store to save space? I assumption to this is yes, as Druid should use bitmap index to convert the string into bit, https://druid.apache.org/docs/latest/design/segments#segment-file-structure. But I am not sure it is also true for nested column.

Within JSON nested columns the dictionaries should be shared across all JSON fields of the same datatype ... so in that case one string dictionary would be used for all three of those key fields ... which means yes you would get space savings.

Hey @John Kowtko, thanks for the rapid response! Does Druid stores the original (raw) JSON somewhere else in the store? I mean if it does that, it still stores three duplicated long string, and it only saves the space for index (bitmap)

I believe it is stored in Smile format, in the raw_compressed section. Here's a meta.smoosh file from the kttm-nested demo datasource .... event, agent, and geo_ip are all nested columns ...

% cat meta.smoosh 
v1,2147483647,1
__time,0,0,1972740
adblock_list,0,33716230,33894228
agent,0,12702266,12702650
agent.__arrayDictionary,0,12709371,12709376
agent.__doubleDictionary,0,12709365,12709371
agent.__field_0,0,16298515,16813982
agent.__field_1,0,16813982,17816309
agent.__field_2,0,17816309,18124470
agent.__field_3,0,18124470,18804622
agent.__field_4,0,18804622,19361424
agent.__field_5,0,19361424,19635843
agent.__longDictionary,0,12709359,12709365
agent.__raw,0,16298510,16298515
agent.__raw_compressed,0,14674019,16298510
agent.__raw_offsets,0,12709376,14674019
agent.__stringDictionary,0,12702650,12709359
app_version,0,33894228,33896539
client_ip,0,19635843,22443881
event,0,7248141,7248538
event.__arrayDictionary,0,7276902,7276907
event.__doubleDictionary,0,7276896,7276902
event.__field_0,0,10119840,10123690
event.__field_1,0,10123690,10827511
event.__field_2,0,10827511,12360166
event.__field_3,0,12360166,12362782
event.__field_4,0,12362782,12686849
event.__field_5,0,12686849,12699877
event.__field_6,0,12699877,12702266
event.__longDictionary,0,7276690,7276896
event.__raw,0,10119835,10119840
event.__raw_compressed,0,9163149,10119835
event.__raw_offsets,0,7276907,9163149
event.__stringDictionary,0,7248538,7276690
geo_ip,0,22443881,22444226
geo_ip.__arrayDictionary,0,22575255,22575260
geo_ip.__doubleDictionary,0,22575249,22575255
geo_ip.__field_0,0,27204853,29218069
geo_ip.__field_1,0,29218069,29589451
geo_ip.__field_2,0,29589451,30293118
geo_ip.__field_3,0,30293118,31909204
geo_ip.__longDictionary,0,22575243,22575249
geo_ip.__raw,0,27204848,27204853
geo_ip.__raw_compressed,0,24501942,27204848
geo_ip.__raw_offsets,0,22575260,24501942
geo_ip.__stringDictionary,0,22444226,22575243
index.drd,0,43684915,43685929
language,0,31909204,31909470
language.__arrayDictionary,0,31913383,31927975
language.__arrayElementDictionary,0,31927975,31929245
language.__arrayElementIndexes,0,32936830,33716230
language.__doubleDictionary,0,31913377,31913383
language.__encodedColumn,0,31929245,32353734
language.__longDictionary,0,31913371,31913377
language.__stringDictionary,0,31909470,31913371
language.__valueIndexes,0,32353734,32936830
loaded_image,0,34744743,35734100
metadata.drd,0,43685929,43687029
number,0,5959101,7248141
path,0,33896539,34744743
referrer,0,35734100,36603826
referrer_host,0,36603826,37117309
screen,0,37365049,38542222
server_ip,0,37117309,37365049
session,0,1972740,5959101
session_length,0,40683844,42488951
timezone,0,42488951,42961620
timezone_offset,0,42961620,43684915
window,0,38542222,40683844
%

Thanks for the details! This is very helpful! IIUC, Smile format does not have compression built-in, that means

raw_compressed

will save three identical long string there and does not save much of the space. Even Druid apply compression algorithm before storing Smile format, the space saving is from compression algorithm's perspective, instead of bitmap's perspective. Am I understand it right? Is there a way to optimize the space here more, like not storing the raw JSON at all? --- What is this field

__raw

do? Does this field stores the raw JSON without any compression? 🙏