In this post, we talk about support for clustering in Splunk. Clustering is about replicating buckets and searchable data for tolerating failures in a distributed environment. There are two configuration settings to aid with replication. One that determines the replication of the raw data and another that determines the replication of searchable data. Both are in the configuration file in the master. Master talks to the peer over HTTP. Peers talk to each other on s2s. The design is such that the peers talk to the master and vice versa but the peers don't need to talk to one another. Basic configuration involves forwarder sending data to peers and search heads talking to both master and peer, Master does most of the management and the peers are the work horses. The hot buckets are created by the indexes but clustering improves the names so as to differentiate them for the nodes. We have a cluster wide bucket id that comprises : index plus id plus guid. We replicate by slices of data in these hot buckets.
We don’t enforce clustering policy on standalone buckets. On each bucket roll, we inform the master. The master keeps track of the states and does ‘fixup’.We schedule a ‘fixup’ on all failures.
Fixup is what happens when a node goes down and we lose the buckets it was working on
Rebuilding was a big problem because it took a lot of time.
Fixup level is broken down into six different levels (streaming, data_safety, generation, replication factor, search factor and checksum)
We schedule the highest priority work at all times.
When peers come up, they get the latest bundle from master
when a cluster node goes down, we could avoid messy state by going offline.
There are two versions of offline -
wait for master to complete (permanent)
second is allow rebalancing primaries by informing master while participating in searches till master gets back to you.
the states are offline->inputs(closed)->wait->done
Primary means there is an in-memory bit mask for that generation.
generation means snap-shoting the states of the primaries across the system.
master tracks which are participating in my current generation
each peer knows which generation it is a primary for.
We don’t enforce clustering policy on standalone buckets. On each bucket roll, we inform the master. The master keeps track of the states and does ‘fixup’.We schedule a ‘fixup’ on all failures.
Fixup is what happens when a node goes down and we lose the buckets it was working on
Rebuilding was a big problem because it took a lot of time.
Fixup level is broken down into six different levels (streaming, data_safety, generation, replication factor, search factor and checksum)
We schedule the highest priority work at all times.
When peers come up, they get the latest bundle from master
when a cluster node goes down, we could avoid messy state by going offline.
There are two versions of offline -
wait for master to complete (permanent)
second is allow rebalancing primaries by informing master while participating in searches till master gets back to you.
the states are offline->inputs(closed)->wait->done
Primary means there is an in-memory bit mask for that generation.
generation means snap-shoting the states of the primaries across the system.
master tracks which are participating in my current generation
each peer knows which generation it is a primary for.
No comments:
Post a Comment