Repository navigation
Implement an M3 reporter - #12
Conversation
| echo "security.provider.1=org.bouncycastle.jce.provider.BouncyCastleProvider" | sudo tee -a /etc/java-7-openjdk/security/java.security | ||
|
|
||
| # Install Thrift 0.9.3 | ||
| sudo apt-get update -qq |
There was a problem hiding this comment.
Why do you need to install thrift binaries? if only for the codegen, might be better to include the generated code in the diff.
There was a problem hiding this comment.
Two options here:
- Yes, installing thrift binaries here are for codegen, but since we compile thrift files as part of the project build, we are also allowing Travis to test our build process as a developer would build the project. However, installing thrift binaries are by far the most time consuming part of a Travis build.
- The other option is that we just check in the generated code, which means we'd have to specify Travis-specific build parameters to not have it try to compile thrift.
Thoughts?
There was a problem hiding this comment.
I don't think developers are going to be regening thrift code often at all, so (1) seems excessive. I suggest (2).
There was a problem hiding this comment.
I've gone ahead and made Thrift code generation skipped by default in addition to checking in the generated source code. Code generation can be triggered by passing a property (documented in DEVELOPER.md).
| @@ -50,15 +48,10 @@ class ScopeImpl implements Scope { | |||
| // performance upside to the consequence of reporting a newly-made metric in | |||
There was a problem hiding this comment.
remove/update the comment to reflect synchronized usage?
|
|
||
| static class Values { | ||
| private ReadWriteLock rwlock = new ReentrantReadWriteLock(); | ||
| private final ReadWriteLock rwlock = new ReentrantReadWriteLock(); |
There was a problem hiding this comment.
why use a rwlock here v synchronise in other places?
There was a problem hiding this comment.
synchronized is used when I would only use one (write) lock. We're making use of ConcurrentHashMap's ability to read values in a thread-safe manner without locks. This rwlock is to protect ArrayList accesses (there isn't a similar Concurrent* version) and having separate read/write locks allow for higher throughput.
There was a problem hiding this comment.
makes sense, could you put a note in the code explaining that intent
| executor = builder.executor; | ||
|
|
||
| addAndRunProcessor(); | ||
| } catch (Exception e) { |
There was a problem hiding this comment.
This try-catch is going to loose Exception type information. E.g. it'll make IOExceptions (e.g. Thrift connection issues) into IllegalArgumentException. Can you avoid doing that?
There was a problem hiding this comment.
Because I hand the Exception e over to my throw, the exception call stack will be preserved, e.g. it'll say IllegalArgumentException "Exception creating M3Reporter" caused by SocketException "something else".
I agree this isn't the prettiest, but an exception here probably signifies an IllegalArgumentException e.g. an incorrect/already used port. We also need to convert these checked exceptions to unchecked exceptions because we can't recover and proceed from these errors.
| protocolFactory = new TBinaryProtocol.Factory(); | ||
| } | ||
|
|
||
| TTransport transport = new TUdpClient(hostPorts[0]); |
There was a problem hiding this comment.
why have hostPorts be an array if you're going to use only one?
There was a problem hiding this comment.
This was done in anticipation of supporting having a transport support multiple host/ports together like the Go version. Listed as issue #11.
| @Override | ||
| public void run() { | ||
| try { | ||
| List<Metric> metrics = new ArrayList<>(freeBytes / 10); |
There was a problem hiding this comment.
how'd you arrive at freeBytes/10?
There was a problem hiding this comment.
per our chat yesterday, seems like this number is just straight from the go implementation.
Could you add an issue to profile the code for different workloads, to gauge expected perf, and to size the defaults/constants correctly. Might be worth applying for YourKit's free OSS license
| return metric; | ||
| } | ||
|
|
||
| private Set<MetricTag> addMetricTagsToSet(Map<String, String> tags) { |
There was a problem hiding this comment.
nit: rename to indicate you're creating a set instead of adding to a provided set (e.g. toMetricTagSet)
| * @param protocol the value to set | ||
| * @return this {@link Builder} with the new value set | ||
| */ | ||
| public Builder protocol(ThriftProtocol protocol) { |
There was a problem hiding this comment.
will users ever override this?
There was a problem hiding this comment.
Yes, they can specify ThriftProtocol.BINARY as opposed to the default TBinaryProtocol.COMPACT, which will make the reporter use that TProtocol accordingly.
There was a problem hiding this comment.
Yeah but will users ever have to set this to the non-default value? I'd rather not reveal the transport mechanism as a choice to the users if we can help it.
There was a problem hiding this comment.
It seems like TCompactProtocol is the more efficient protocol here, so I'll just default to using that.
| TimerValue timerValue = resourcePool.getTimerValue(); | ||
| timerValue.setI64Value(iValue); | ||
| metricCopy.getMetricValue().setTimer(timerValue); | ||
| break; |
| @@ -189,17 +194,23 @@ public int compareTo(Duration other) { | |||
| public String toString() { | |||
There was a problem hiding this comment.
could you put a javadoc explaining the expected output
| } else if (nanos == Long.MIN_VALUE) { | ||
| // Return hard coded response as workaround because | ||
| // Math.abs(Long.MIN_VALUE) == Long.MIN_VALUE | ||
| return "-2562047h47m16.854775808s"; |
There was a problem hiding this comment.
this feels hacky, why do you need the special case?
|
|
||
| StringBuilder buf = new StringBuilder(16); | ||
| // 25 capacity from max String length produced from Long.MIN_VALUE nanos | ||
| StringBuilder buf = new StringBuilder(25); |
There was a problem hiding this comment.
could you make a static final for 25
|
|
||
| private final MetricBatch metricBatch = new MetricBatch(); | ||
|
|
||
| private final Object calcLock = new Object(); |
There was a problem hiding this comment.
just occurred to me, can't you use the object itself for locking? as in why have an additional member called calcLock when you can synchronise on calc
There was a problem hiding this comment.
calcLock protects both calc and calcProtocol. I'll put in a comment.
| return; | ||
| } | ||
|
|
||
| TException exception = null; |
There was a problem hiding this comment.
considering you're just logging, does this need to be an exception still?
There was a problem hiding this comment.
Nope, will remove and just log.
| int tries = 0; | ||
| boolean tryFlush = true; | ||
|
|
||
| synchronized (metricBatch) { |
There was a problem hiding this comment.
Does making metricBatch an instance member instead of alloced/cleaned up in the flush call buy you anything?
There was a problem hiding this comment.
Good point. Made metricBatch a local variable and add commonTags to it instead - as discussed offline, should help with throughput as this removes the synchronized block.
| // Holds metrics to be flushed | ||
| private final BlockingQueue<SizedMetric> metricQueue; | ||
| // The service used to execute metric flushes | ||
| private ExecutorService executor = Executors.newFixedThreadPool(THREAD_POOL_SIZE); |
There was a problem hiding this comment.
nit: make this configurable in the builder.
| public void run() { | ||
| try { | ||
| while (!executor.isShutdown()) { | ||
| // This `take` call will block until there is an item on the queue to take. |
There was a problem hiding this comment.
nit: update this comment to indicate why you're using poll instead of take
| } | ||
| } | ||
| } catch (InterruptedException e) { | ||
| Thread.currentThread().interrupt(); |
There was a problem hiding this comment.
Sorry I'm missing something. A processor is already running on a different thread, and it's received an interrupt, why does it need to interrupt itself again?
There was a problem hiding this comment.
Oops good catch - missed this earlier.
| // catch block. | ||
| SizedMetric sizedMetric = metricQueue.poll(MAX_PROCESSOR_WAIT_UNTIL_FLUSH_MILLIS, TimeUnit.MILLISECONDS); | ||
|
|
||
| if (sizedMetric != null) { |
There was a problem hiding this comment.
could you restructure for readability:
if (sizedMetric != null) {
flushMetricInteration(sizedMetric);
continue;
}
// If we don't get any new metrics after waiting the specified time,
// flush what we have so far.
if (!pendingMetrics.isEmtpy()) {
flushMetricIteration(SizedMetric.FLUSH);
}
|
|
||
| // Important to use `shutdownNow` instead of `shutdown` to interrupt processor | ||
| // thread(s) or else they will block forever | ||
| executor.shutdownNow(); |
There was a problem hiding this comment.
i think you need to re-order: i.e. isOpen = false; before this, no?
|
|
||
| @Override | ||
| public void reportCounter(String name, Map<String, String> tags, long value) { | ||
| synchronized (isOpenLock) { |
There was a problem hiding this comment.
Does this (synchronized on isOpenLock) affect throughput much?
There was a problem hiding this comment.
As discussed offline, removed the need for an isOpen flag. Instead, use a sentinel metric in the queue to indicate that all metrics coming after it was put in after reporter was closed.
| BucketPair[] bucketPairs = BucketPairImpl.bucketPairs(buckets); | ||
|
|
||
| if (tags == null) { | ||
| tags = new HashMap<>(2, 1); |
There was a problem hiding this comment.
what does the (2,1) here mean?
There was a problem hiding this comment.
Removed this optimization for readability.
| } | ||
| } | ||
|
|
||
| private class Processor implements Runnable { |
There was a problem hiding this comment.
Could you break this class out into a separate file? feels like there's too much going on in this file
There was a problem hiding this comment.
After attempting this, I found that the Processor is too tightly integrated to the M3Reporter to meaningfully have it broken out of it. The shared metrics queue, flushing logic, and synchronization variables will probably not enable code reuse if this was split up.
|
|
||
| if (tries < maxTries) { | ||
| try { | ||
| Thread.sleep(FLUSH_RETRY_DELAY_MILLIS); |
There was a problem hiding this comment.
try catch in a try catch feels like smell, can you that clean up
There was a problem hiding this comment.
Agreed that this isn't the prettiest code, but I want to sleep only when I hit an exception, and sleep throws a checked exception 😕. Extracting out just a sleep function seems a little silly.
There was a problem hiding this comment.
As discussed offline, there seems to be little upside to having this retry logic in the first place - removing this for now.
| BucketPair[] bucketPairs = BucketPairImpl.bucketPairs(buckets); | ||
|
|
||
| if (tags == null) { | ||
| tags = new HashMap<>(2, 1); |
There was a problem hiding this comment.
use const vars rather than literals heres
There was a problem hiding this comment.
These numbers are used for their specific circumstances. If tags == null, then we know that there will only be two items in the map (the two that we add later on). The 2 is the initial capacity of the buckets, and the 1 defines when to rehash.
Added a comment explaining this.
| return; | ||
| } | ||
|
|
||
| boolean tryFlush = true; |
There was a problem hiding this comment.
nit: i don't think you need this var or the while loop wrapping the try below anymore
| // When this reporter is closed, shutdownNow will be called on the executor, | ||
| // which will interrupt this thread and proceed to the `InterruptedException` | ||
| // catch block. | ||
| SizedMetric sizedMetric = metricQueue.poll(MAX_PROCESSOR_WAIT_UNTIL_FLUSH_MILLIS, TimeUnit.MILLISECONDS); |
There was a problem hiding this comment.
could you make this configurable, and change the default to 10s (default reporting interval in go-tally iirc)
|
|
||
| if (sizedMetric == SizedMetric.FLUSH) { | ||
| flush(pendingMetrics, commonTags); | ||
|
|
| int size = sizedMetric.getSize(); | ||
|
|
||
| if (metricsSize + size > freeBytes) { | ||
| flush(pendingMetrics, commonTags); |
There was a problem hiding this comment.
A question for you ~ what happens if we get a lot of timer values emitted in a short burst. E.g. the current java client runs into this with metrics for a Kafka topic being processed.
I'm afraid of this - we'd queue all the timers directly into the metricsQueue, because there's no buffering. If the processor thread doesn't get scheduled during the queuing, we'd fill the queue, and be unable to flush because we can't queue the sentinel flush metric. Can this happen?
There was a problem hiding this comment.
talked offline, not a deadlock because we can still make forward progress despite not being able to enqueue a flush. We'd drop metrics if the queue was full but that's an accepted semantic already.
Implement an M3 reporter. Additional tests will come in a separate PR to reduce PR size.