While this is true, things were also achieved with lots of experimentation, both on the structural side and on materials applications. That knowledge was vital to have good practices on what to use in each situation and predict maintenance and consequences in extreme conditions. The same applies to the conception of fault-tolerant reliable systems to process large volumes of data.

Normal approaches start with mere reasonable assumptions on data volume, ingestion rate, document size, but others are hard to predict such as processing time, indexing time, etc. It really helps to know how components behave under pressure and calculate what are the limits of the current design.

Many (self-entitled) architects like to take the designing process as cooking a big blend of hype-based technologies and it’s very easy to get burnt. Key factors such as SLAs, peak times and hardware limitations greatly affect which components you choose to put in the pan and how would you mix them together.

The only way to know how certain software components behave, they have to be exercised with a great volume of data, similar to the one they will process once in production. Data that can be just fed from the productions servers if it exists and if the infrastructure allows. More often than not, the data is not yet available in the target formats and there’s the necessity of trying out with generated data in the chosen formats.

Whenever I want to test and benchmark the systems I’m working on, I use a synthetic data generator called log-synth. Log synth is one of those swiss-army knives for data generation. It has plenty of generators, based on parametric statistical methods. The good part is that it’s open-source and can be easily extended with new generating algorithms and output formats.

The most common output formats are JSON and CSV.

Recently I’ve added a template based generator that extends the range of available output formats. It leverages the data generation algorithms that ship with it and feeds that into a templated document using Freemarker templating language.

To generate a sample vCard simply create a file template.txt:

1
2
3
4
5
6
7
8
9
10
11

BEGIN:VCARD
VERSION:3.0
N:${last_name.asText()};${first_name.asText()};;${title.asText()}
ORG:Sample Org
TITLE:${title.asText()}
PHOTO;VALUE=URL;TYPE=GIF:http://thumbs.example.com/${filename.asText()}/${first_name.asText()?lower_case  }.gif
TEL;TYPE=HOME,VOICE:${phone_number.asText()}
ADR;TYPE=WORK:;;${address.asText()?split(" ")?join(";")}
EMAIL;TYPE=PREF,INTERNET:${first_name.asText()[0]?lower_case}${last_name.asText()?lower_case}@example.com
REV:${first_visit.asText()}
END:VCARD

Then a schema file, let’s call it schema.txt:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

[
    {"name":"title", "class":"string", "dist":{"Mr":0.5, "Mrs.":0.14, "Miss":0.36}},
    {"name":"first_name", "class":"name", "type":"first"},
    {"name":"last_name", "class":"name", "type":"last"},

  {"name": "filename", "class": "join", "separator": "/", "value": {
          "class":"sequence",
          "length":2,
          "array":[
              {"class":"string", "dist":{"small":10, "medium":5, "large":2}},
              {"class":"string", "dist":{"high":10, "low":5, "mobile":15}}
          ]
      }
  },

  {"name": "phone_number", "class": "join", "separator": "-", "value": {
          "class":"sequence",
          "length":3,
          "array":[
              { "class": "int", "min": 100, "max": 999},
              { "class": "int", "min": 100, "max": 999},
              { "class": "int", "min": 100, "max": 999}
          ]
      }
  },

    {"name":"address", "class":"address"},
    {"name":"first_visit", "class":"date", "format":"yyyy-MM-dd HH:mm:ssZ"}
]

To invoke the log-synth, just do:

java -cp .:./target/log-synth-0.1-SNAPSHOT-jar-with-dependencies.jar com.mapr.synth.Synth -count 5000 -schema schema.txt -template template.txt -format TEMPLATE -output output/

The output documents will end up in the output/ folder as expected and they will look like:

1
2
3
4
5
6
7
8
9
10
11

BEGIN:VCARD
VERSION:3.0
N:Kittle;Gwendolyn;;Mr
ORG:Sample Org
TITLE:Mr
PHOTO;VALUE=URL;TYPE=GIF:http://thumbs.example.com/small/mobile/gwendolyn.gif
TEL;TYPE=HOME,VOICE:774-383-580
ADR;TYPE=WORK:;;18033;Quaking;Brook;Avenue
EMAIL;TYPE=PREF,INTERNET:gkittle@example.com
REV:2013-07-14 01:37:08+0100
END:VCARDBEGIN:VCARD

I invite you to explore this very handy tool on Github.

Most of the common aggregation operations like counting and summing can be performed in parallel, as long there is a reduce phase where the result on each node can be combined. However, this is not very trivial for calculating histograms, as we need all the data on one dimension so that we can represent it in an histogram.

Having the data being processed by multiple nodes, each node is only able to construct an histogram of the partial data it receives. Ben-Haim and Tom-Tov presented a solution that uses an heap-based data structure to represent the data and a merge algorithm that allows to merge the data structures computed on different nodes into one that is an approximate histogram of all the dataset.

This technique has been applied by MetaMarkets with good accuracy for most of what an histogram can tell us about the data distribution: calculating the average and counting the quartiles and total number of data/events.

I took the liberty of doing a simple implementation of it, that is now being used in production for some months now:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234

import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Sets;

import java.util.Iterator;
import java.util.Set;
import java.util.TreeSet;

/**
 * This is an approximate histogram class
 */
public class ApproximateHistogram {
    private final int numPairs;
    private final TreeSet<CentroidPair> heap;

    public ApproximateHistogram(int numPairs) {
        this.numPairs = numPairs;
        this.heap = new TreeSet();
    }

    public ApproximateHistogram(TreeSet<CentroidPair> heap, int numPairs) {
        this.numPairs = numPairs;
        this.heap = heap;
    }

    public Set<CentroidPair> heap() {
        return ImmutableSet.copyOf(heap);
    }

    public void update(CentroidPair p) {
        Iterator<CentroidPair> it = heap.iterator();
        while (it.hasNext()) {
            CentroidPair cp = it.next();

            int compare = Double.compare(cp.centroid, p.centroid);
            if (compare == 0) {
                cp.count += p.count;
                return;
            } else if (compare == 1) {
                break;
            }
        }

        // there was no similar centroid, so let's add the point to the heap
        heap.add(p);

        compress();
    }

    private void compress() {
        if (heap.size() <= numPairs) {
            return; // compress only if needed
        }

        int i = 0;
        double minDiff = Double.MAX_VALUE;
        CentroidPair last = null, lastLast = null;

        // [ ..., minA, minB, ... ] two consecutive pairs which centroid diff is the minimum
        CentroidPair minA = null, minB = null;

        Iterator<CentroidPair> it = heap.iterator();
        while (it.hasNext()) {
            lastLast = last;
            last = it.next();

            if (i > 0) {
                double diff = last.centroid - lastLast.centroid;

                if (diff < minDiff) {
                    minA = lastLast;
                    minB = last;
                    minDiff = diff;
                }
            }
            ++i;
        }

        int repCount = Math.abs(minA.count) + Math.abs(minB.count);
        double repCentroid = (minA.centroid * Math.abs(minA.count) + minB.centroid * Math.abs(minB.count)) / repCount;
        CentroidPair replacementPair = new CentroidPair(-repCount, repCentroid); // store with negative sign the compressed entries
        heap.remove(minA);
        heap.remove(minB);
        heap.add(replacementPair);
    }


    public static ApproximateHistogram merge(ApproximateHistogram... histograms) {
        ApproximateHistogram merged = histograms[0];

        for (int i = 1; i < histograms.length; i++) {
            merged = merge(merged, histograms[i]);
        }

        return merged;
    }

    public static ApproximateHistogram merge(ApproximateHistogram a, ApproximateHistogram b) {
        int biggestSize = a.heap.size();

        if (b.heap.size() > biggestSize) {
            biggestSize = b.heap.size();
        }

        TreeSet<CentroidPair> mergedHeap = Sets.newTreeSet();
        mergedHeap.addAll(a.heap);
        mergedHeap.addAll(b.heap);

        final ApproximateHistogram merged = new ApproximateHistogram(mergedHeap, biggestSize);

        // add the centroids of B to the merged (ignoring compression)
        int compressTimes = mergedHeap.size() - biggestSize;

        while (compressTimes-- > 0) {
            merged.compress();
        }

        return merged;
    }


    public double countBelow(double cutPoint) {
        final double EPSILON = 0.00000001;
        if (heap.isEmpty()) return 0.0;


        CentroidPair[] heapPoints = heap.toArray(new CentroidPair[heap.size()]);

        int j = 0;
        for (int i = 0; i < heapPoints.length; i++) {
            int count = heapPoints[i].count;
            double diff = heapPoints[i].centroid - cutPoint;

            // there's a pair with the cutPoint as centroid
            if (Math.abs(diff) < EPSILON) {
                return j + ((count > 0) ? count : Math.abs(count) / 2.0);
            } else if (diff > 0) {
                // we already passed. it's somewhere between the last and this one

                // CASE: the cutPoint is before the first centroid point
                if (i == 0) {
                    if (count > 0) return 0.0; // we are sure no entry was less than the first centroid
                    return Math.abs(count) * cutPoint / (2.0 * heapPoints[i].centroid);// the first pair is an average. do the calculation
                }

                CentroidPair lastPoint = heapPoints[i - 1];
                CentroidPair currentPoint = heapPoints[i];
                int lastCount = lastPoint.count;

                // if the last point is just an average point, discount it
                j -= ((lastCount < 0) ? Math.abs(lastCount) : 0); // WHT?

                lastCount = Math.abs(lastCount);

                double mb = lastCount + (Math.abs(currentPoint.count) - lastCount) * (cutPoint - lastPoint.centroid) / (currentPoint.centroid - lastPoint.centroid);
                double sum = (lastCount + mb) * (cutPoint - lastPoint.centroid) / (2.0 * (currentPoint.centroid - lastPoint.centroid));

                return sum + j + Math.abs(lastCount) / 2.0;
            }

            j += Math.abs(count);
        }

        // some logic for the cases where b > centroid[last]
        CentroidPair lastPoint = heapPoints[heapPoints.length - 1];
        int count = lastPoint.count;

        // last point is an average and there's more than one
        if (count < 0 && heapPoints.length > 1) {
            count = Math.abs(count);
            CentroidPair lastLastPoint = heapPoints[heapPoints.length - 2];

            // calculate a virtual final point which is separated half the distance than the last one
            double distanceToPreviousPoint = lastPoint.centroid - lastLastPoint.centroid;
            distanceToPreviousPoint /= 4.0;
            double finalCentroid = lastPoint.centroid + distanceToPreviousPoint;

            double diff = finalCentroid - cutPoint;

            // count all!
            if (diff > 0) {
                j -= count / 2.0;

                double trapezoidSum = count * (cutPoint - lastPoint.centroid) / (2.0 * distanceToPreviousPoint);

                return j + trapezoidSum;
            }
            // else return j!
        }


        return j;
    }

    public double avg() {
        int count = 0;
        double sum = 0.0;

        for (CentroidPair centroidPair : heap) {
            int absCount = Math.abs(centroidPair.count);
            count += absCount;
            sum += absCount * centroidPair.centroid;
        }

        return (count > 0) ? sum / count : 0.0;
    }

    public int count() {
        int sum = 0;
        for (CentroidPair centroidPair : heap) {
            sum += centroidPair.count;
        }
        return sum;
    }

    public static class CentroidPair implements Comparable<CentroidPair> {
        int count;
        double centroid;

        public CentroidPair(int count, double centroid) {
            this.count = count;
            this.centroid = centroid;
        }

        @Override
        public int compareTo(CentroidPair o) {
            return Double.compare(this.centroid, o.centroid);
        }

        @Override
        public String toString() {
            return new StringBuilder("(").append(count).append(", ").append(centroid).append(")").toString();
        }
    }
}

Internally the histogram is represented by a set of points (count, centroid), ordered by it’s centroid. When a new point is added, if the centroid already exists we increase the count number, otherwise we add the point with count 1 to the list.

Each histogram has a limit of points to keep and when a new insert exceeds this limit, a compression takes place. The compression consists in merging the two consecutive points where the difference between its centroids is the lower. The two are replaced by a single point with centroid on a place nearer to the neighbor point that has more counts: if they have the same count, it would be on the middle. The count of the new point will be the sum of the two old ones.

As Java doesn’t have unsigned numeric types, this implementation exploits the signal in the count field to flag if that point has been originated from compression of two other or if it is from raw observations. This can help answering to questions like: how many values are below X? If the points have a positive count for every point whose centroid is below X, we can truly count them. If they are negative, we know that point is an approximation, so we calculate the count using the trapezoidal estimation of Ben-Haim and Tom-Tov. This gives more accurate results than assuming every point might be an approximation and requires no extra space in Java-based data structures.

For merging more than one histogram, which happens when we want to combine results computed on different nodes. This is done by creating a big heap with the combined values of the histograms and applying compression on that heap, as described above, until the heap has the maximum number of points.

For very disperse data, this data structure may yield bad approximations if the number of points is not high enough. This data structure is very flexible and it’s easy to use it for streams with different distributions by just tuning the number of centroids we keep.

Online group-by operations

As mentioned on part one, each consumer will receive messages from a set of partitions in a way that each partition will have only one consumer. The subscriber process can then have a consumer stream per thread and can co-operate with other instances running on other machines by defining each consumer to belong to the same consumer group. There no gain on having the total number of threads in the consumer cluster higher than the number of partitions, as each partition will comunicate at most through one consumer stream.

Each consumer thread receives a message from a partition. The message is parsed into a POJO that will be a key to a counting hash map.

1
2
3
4
5
6
7

ConsumerIterator<byte[], byte[]> it = stream.iterator();

while (it.hasNext()) {
  String value = new String(it.next().message());
  final String[] tokens = value.split(";", -1);

  // ...

The example above shows how to parse a message, assuming it’s a comma-separated value string. To create a compatible system with the existing log based solution and for debugging purposes, we first adopt the old CSV as message format. However, other solutions exist such as Kryo, Avro and Thrift.

The parsed POJO is the counter key and it has a custom hashCode method implementation. The hashCode is invoked by the Java’s HashMap and it uses the returning value to locate the value internally. When more than one entry exists with the same hashCode, the equals method is used to determine the equality to the query key.

To aggregate the counts by a set of attributes, we can use the hashCode and equals methods. These methods must have a consistent behaviour considering two objects to be equal if the set of group-by attributes have identical values in both. The hashCode calculation has to take this values into account, as shown in the snippet below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

import org.apache.commons.lang3.builder.EqualsBuilder;
import org.apache.commons.lang3.builder.HashCodeBuilder;

public class SaleEventByHourAndCategory extends SaleEvent {
  static final TimeZone GMT_TZ = TimeZone.getTimeZone("GMT:00");

  protected Long getHour() {
      // consider move this to a Util class
      Calendar calendar = Calendar.getInstance(GMT_TZ);
      calendar.setTime(new Date(getUnixTimestamp()));

      calendar.set(Calendar.MINUTE, 0);
      calendar.set(Calendar.SECOND, 0);
      calendar.set(Calendar.MILLISECOND, 0);

      return Long.valueOf(calendar.getTime().getTime());
    }

  @Override
  public int hashCode() {
      return new HashCodeBuilder()
          .append(getHour())
          .append(getCategoryId())
          .hashCode();
  }

  @Override
  public boolean equals(Object obj) {
      if (obj instanceof SaleEventByHourAndCategory == false) {
          return false;
      } if (this == obj) {
          return true;
      }

      final SaleEventByHourAndCategory otherObject = (SaleEventByHourAndCategory) obj;

      return new EqualsBuilder()
          .append(getHour(), otherObject.getHour())
          .append(getCategoryId(), otherObject.getCategoryId())
          .isEquals();
  }
}

The implementation of hashCode and equals use the Apache Commons Lang library.

Counter Map Implementations

Each consumer thread will count the events in a shared data structure, to avoid spending too much memory with multiple duplicated entries in each thread’s counter map. ConcurrentHashMap is a native JVM implementation that aims to reduce contention by splitting the key space in blocks. Each block will have a lock to control the write accesses.

Other alternative HashMap implementation is the NonBlockingHashMap, written by Azul System’s Dr. Cliff Click, is an implementation that doesn’t assure full consistency ruturning only the result of the last completed update operation. This behaviour is lock free and leverage compare and set/swap operations to allow multiple update operations. A benchmark between both implementations can be found here.

Aggregated Result Flushing

The computed operations, either counts, sums or averages are hold in each subscriber process’s memory and must be flushed to a proper persistent storage. We decided to continue with MySQL to avoid porting all the existent visualization tools. In a scenario of multiple instances of the subscriber process, each instance will have a partial result of the aggregated operation. If there are two processes with 20 threads each, each one will have the count for the messages received from half the partitions. Because they only know part of the result, flushing must be done using an UPDATE SQL query or an INSERT in the case of non-existence of that key. To save the check of existence, an INSERT ON DUPLICATE KEY UPDATE is used.

In the first experiments, the subscriber processes flushed the in-memory counts to the same database. However, some of these subscribers were consuming from a broker in Europe region and had to upload the results to a US datacenter-based MySQL instance. To optimize the flow, all the subscribers will act as producers and will publish their partial results into a separate topic. This topic will be read by a single flusher process.

This change of approach improved the throughtput of MySQL ingestion, because less write transations fail. The flusher process executes transaction after transaction in batches of 2000 queries and has a consistent per batch execution time. The partial results publishing is done to the US broker using Kryo to serialize and Snappy to compress. This attempts to reduce both the bandwidth required and the latency time.

In this post, I’ve covered how to do group-by aggregation operations in-memory and temporarily flush them to a persistent engine, such as MySQL. In the next post I’ll cover infrastructure matters and deployment of the Kafka stack in AWS.

I am working in real-time advertisement industry and most of the reports generated require processing huge amounts of data. The reports are essentially counting events for traffic estimation or classification. However, counting a thousand events per second, in a distributed environment might not be as trivial as it seems.

The old process ingested hourly logs, captured from some dozens of nodes every hour. Each node uploaded a compressed bulk of log files to AWS S3, and Hadoop jobs triggered every hour had to load them up again into HDFS. The whole time spent in this IO-dependent ETL process was considerably huge and the increasing amount of data was causing a delay of several hours to get reporting metrics ready.

In real-time bidding, the reporting data of some hours ago might become totally useless for the current bidding decisions, as the market and the opportunities in inventory vary a lot, depending on the countries you serve ads and depending also on the time of the day.

The latency and the time-based utility of reporting data motivated the change into an event-driven pipeline.

Requirements

In order to keep business running, the new system had to be implemented in parallel with the batch pipeline. The first change was to gather data directly from each node into a message broker and have a set of consumers processing the data.

In the first phase, messages could be simply gathered with a tail -f in the log files and the output piped with a message publisher application that reads them from the input stream.

The message queue system had to be able to deliver messages with high-throughput rather than low-latency and had to persist them to disk, as it was meant to replace the file based logging pipeline. Logging to network inside a datacenter can be faster than logging to disk (see Latency numbers every programmer should know, by Peter Norvig and Jeff Dean).

Message global ordering wasn’t really a requirement for the volume estimation processing, as each opportunity event contains a timestamp we can use to group counts by.

For volume estimation, data loss up to some seconds is also acceptable in a failure scenario, as even if we miscount some events, most of them get will be counted.

Apache Kafka

After analysing the most popular, open-source available queueing solutions I could conclude that most of them focused on low-latency message deliver and just a few supported persistence. Apache Kafka was the one that seemed to match our requirements.

Kafka started as a LinkedIn in-house project, developed when the company was passing through a similar period of transition when pure batch processing started to affect the feedback loop and become a pain to the business.

Kafka is a distributed message publishing/subscribing system of one or more brokers, each one with a set of zero or more partitions for each existing topic. Kafka persists periodically messages to disk, so in case of failure the last ones might get loss. This speeds up the publishing operation, as publishers don’t need to wait until that data gets written to disk.

When a publisher connects to a Kafka cluster, it queries which partitions exist for the topic and which nodes are responsible for each partition.

Each publisher then acts like a card dealer in a poker game, handing messages to partitions as if they were cards. They assign messages to each partition using an hashing algorithm and deliver them to the broker responsible for that partition.

For each partition, a broker stores the incoming messages with monoticaly increasing order identifiers and persists the “deck” to disk using a data structure with access complexity of O(1).

A subscriber is a set of co-operating processes that belong to a consumer-group. Each consumer in the group get assigned a set of partitions to consume from. One key difference to other message queue systems is that each partition is consumed by the same consumer and this allows each consumer to track their progress on consumption on the thread and update it asynchronously.

This simplifies the subscription process as the consumer doesn’t need to reply with acknowledgement responses to each “card” it gets from the partition “deck” and the broker doesn’t need to store for each message, if it was processed or not. Consumers keep track on what they consume and store asynchronously in Zookeeper. This is the key point that allows high-throughput. In case of consumer failure, a new process can start from the last saved point, eventually processing the last messages twice.

The broker subscription API requires the identifier of the last message a consumer had from a given partition and starts to stream from that point on. The constant access time data structures on disk play an important role here to reduce disk seeks.

Both consumer groups and brokers are dynamic, so if the amount of incoming messages increase, you can just add new broker nodes to the list and each of them will contain a defined number of partitions for each topic. According to the number of partitions you have, you can also spawn more subscriber processes if the ones you have can’t handle the new partition’s messages in a reasonable time.

This kind of flexibility by design reduces the random IO in the broker machines and make the whole Kafka system a very stable one in heavy-load production environments.

In the part II of the series, I will write how consumers perform the distributed count of events.