1 |
Sack - sharding memory hash in perl |
2 |
|
3 |
Main design goal is to have interactive environment to query |
4 |
perl hashes which are bigger than memory on single machine. |
5 |
|
6 |
It implemented using TCP sockets between perl processes. |
7 |
This allows horizontal scalability both on multi-core machines |
8 |
as well as across the network to additional machines. |
9 |
|
10 |
Reading data into hash is done using any perl module which |
11 |
returns perl hash and supports offset and limit to select just |
12 |
subset of data (this is required to create disjunctive shards). |
13 |
|
14 |
Views are small perl snippets which are called for each record |
15 |
on each shard with $rec. Views create data in $out hash which |
16 |
is automatically merged in output. |
17 |
|
18 |
You can influence default shard merge by adding " +" (space plus) |
19 |
to name of your key to indicate that values should be summed |
20 |
after collecting them from from shards. |
21 |
|