Makes use of dictionary indexes for strings if any.
Depends only on the presence of dictionary per batch of rows (where the batch
must be substantially greater than its dictionary for optimization to help).
For single column hash maps (groups or joins), it can be turned into a flat
indexed array instead of a map. Create an array of class objects as stored
in ObjectHashSet having the length same as dictionary so that dictionary
index can be used to directly lookup the array. Then for the first lookup
into the array for a dictionary index, lookup the actual ObjectHashSet
for the key to find the map entry object and insert into the array.
An alternative would be to pre-populate the array by making one pass through
the dictionary, but it may not be efficient if many of the entries in the
dictionary get filtered out by query predicates and never need to consult
the created array.
For multiple column hash maps having one or more dictionary indexed columns,
there is slightly more work. Instead of an array as in single column case,
create a new hash map where the key columns values are substituted by
dictionary index value. However, the map entry will remain identical to the
original map so to save space add the additional index column to the full
map itself. As new values are inserted into this hash map, lookup the full
hash map to locate its map entry, then point to the same map entry in this
new hash map too. Thus for subsequent look-ups the new hash map can be used
completely based on integer dictionary indexes instead of strings.
An alternative approach can be to just store the hash code arrays separately
for each of the dictionary columns indexed identical to dictionary. Use
this to lookup the main map which will also have additional columns for
dictionary indexes (that will be cleared at the start of a new batch).
On first lookup for key columns where dictionary indexes are missing in
the map, insert the dictionary index in those additional columns.
Then use those indexes for equality comparisons instead of string.
The multiple column dictionary optimization will be useful for only string
dictionary types where cost of looking up a string in hash map is
substantially higher than integer lookup. The single column optimization
can improve performance for other dictionary types though its efficacy
for integer/long types will be reduced to avoiding hash code calculation.
Given this, the additional overhead of array maintenance may not be worth
the effort (and could possibly even reduce overall performance in some
cases), hence this optimization is currently only for string type.
Makes use of dictionary indexes for strings if any. Depends only on the presence of dictionary per batch of rows (where the batch must be substantially greater than its dictionary for optimization to help).
For single column hash maps (groups or joins), it can be turned into a flat indexed array instead of a map. Create an array of class objects as stored in ObjectHashSet having the length same as dictionary so that dictionary index can be used to directly lookup the array. Then for the first lookup into the array for a dictionary index, lookup the actual ObjectHashSet for the key to find the map entry object and insert into the array. An alternative would be to pre-populate the array by making one pass through the dictionary, but it may not be efficient if many of the entries in the dictionary get filtered out by query predicates and never need to consult the created array.
For multiple column hash maps having one or more dictionary indexed columns, there is slightly more work. Instead of an array as in single column case, create a new hash map where the key columns values are substituted by dictionary index value. However, the map entry will remain identical to the original map so to save space add the additional index column to the full map itself. As new values are inserted into this hash map, lookup the full hash map to locate its map entry, then point to the same map entry in this new hash map too. Thus for subsequent look-ups the new hash map can be used completely based on integer dictionary indexes instead of strings.
An alternative approach can be to just store the hash code arrays separately for each of the dictionary columns indexed identical to dictionary. Use this to lookup the main map which will also have additional columns for dictionary indexes (that will be cleared at the start of a new batch). On first lookup for key columns where dictionary indexes are missing in the map, insert the dictionary index in those additional columns. Then use those indexes for equality comparisons instead of string.
The multiple column dictionary optimization will be useful for only string dictionary types where cost of looking up a string in hash map is substantially higher than integer lookup. The single column optimization can improve performance for other dictionary types though its efficacy for integer/long types will be reduced to avoiding hash code calculation. Given this, the additional overhead of array maintenance may not be worth the effort (and could possibly even reduce overall performance in some cases), hence this optimization is currently only for string type.