C# Class KafkaNet.Producer

Provides a simplified high level API for producing messages on a topic.
Inheritance: KafkaNet.CommonQueries
Mostrar archivo Open project: gigya/KafkaNetClient Class Usage Examples

Public Methods

Method Description
Dispose ( ) : void
GetTopicFromCache ( string topic ) : KafkaNet.Protocol.Topic

Get the metadata about a given topic.
GetTopicOffsetAsync ( string topic, int maxOffsets = 2, int time = -1 ) : Task>
Producer ( IBrokerRouter brokerRouter, int maximumAsyncRequests = MaximumAsyncRequests, int maximumMessageBuffer = MaximumMessageBuffer ) : KafkaNet.Common

Construct a Producer class.

The maximumAsyncRequests parameter provides a mechanism for minimizing the amount of async requests in flight at any one time by blocking the caller requesting the async call. This affectively puts an upper limit on the amount of times a caller can call SendMessageAsync before the caller is blocked. The MaximumMessageBuffer parameter provides a way to limit the max amount of memory the driver uses should the send pipeline get overwhelmed and the buffer starts to fill up. This is an inaccurate limiting memory use as the amount of memory actually used is dependant on the general message size being buffered. A message will start its timeout countdown as soon as it is added to the producer async queue. If there are a large number of messages sitting in the async queue then a message may spend its entire timeout cycle waiting in this queue and never getting attempted to send to Kafka before a timeout exception is thrown.

SendMessageAsync ( Message messages, string topic, int partition, Int16 acks = 1 ) : Task
SendMessageAsync ( string topic ) : Task
SendMessageAsync ( string topic, IEnumerable messages, Int16 acks = 1, System.TimeSpan timeout = null, MessageCodec codec = MessageCodec.CodecNone, int partition = null ) : Task

Send an enumerable of message objects to a given topic.
SendMessageAsync ( string topic, int partition ) : Task
Stop ( bool waitForRequestsToComplete = true, System.TimeSpan maxWait = null ) : void

Stops the producer from accepting new messages, and optionally waits for in-flight messages to be sent before returning.

Private Methods

Method Description
BatchSendAsync ( ) : Task
IsNotDisposedOrHasMessagesToProcess ( ) : bool
ProduceAndSendBatchAsync ( List messages, CancellationToken cancellationToken ) : Task
SetResult ( List sendTasks ) : Task

Method Details

Dispose() public method

public Dispose ( ) : void
return void

GetTopicFromCache() public method

Get the metadata about a given topic.
public GetTopicFromCache ( string topic ) : KafkaNet.Protocol.Topic
topic string The name of the topic to get metadata for.
return KafkaNet.Protocol.Topic

GetTopicOffsetAsync() public method

public GetTopicOffsetAsync ( string topic, int maxOffsets = 2, int time = -1 ) : Task>
topic string
maxOffsets int
time int
return Task>

Producer() public method

Construct a Producer class.
The maximumAsyncRequests parameter provides a mechanism for minimizing the amount of async requests in flight at any one time by blocking the caller requesting the async call. This affectively puts an upper limit on the amount of times a caller can call SendMessageAsync before the caller is blocked. The MaximumMessageBuffer parameter provides a way to limit the max amount of memory the driver uses should the send pipeline get overwhelmed and the buffer starts to fill up. This is an inaccurate limiting memory use as the amount of memory actually used is dependant on the general message size being buffered. A message will start its timeout countdown as soon as it is added to the producer async queue. If there are a large number of messages sitting in the async queue then a message may spend its entire timeout cycle waiting in this queue and never getting attempted to send to Kafka before a timeout exception is thrown.
public Producer ( IBrokerRouter brokerRouter, int maximumAsyncRequests = MaximumAsyncRequests, int maximumMessageBuffer = MaximumMessageBuffer ) : KafkaNet.Common
brokerRouter IBrokerRouter The router used to direct produced messages to the correct partition.
maximumAsyncRequests int The maximum async calls allowed before blocking new requests. -1 indicates unlimited.
maximumMessageBuffer int The maximum amount of messages to buffer if the async calls are blocking from sending.
return KafkaNet.Common

SendMessageAsync() public method

public SendMessageAsync ( Message messages, string topic, int partition, Int16 acks = 1 ) : Task
messages KafkaNet.Protocol.Message
topic string
partition int
acks System.Int16
return Task

SendMessageAsync() public method

public SendMessageAsync ( string topic ) : Task
topic string
return Task

SendMessageAsync() public method

Send an enumerable of message objects to a given topic.
public SendMessageAsync ( string topic, IEnumerable messages, Int16 acks = 1, System.TimeSpan timeout = null, MessageCodec codec = MessageCodec.CodecNone, int partition = null ) : Task
topic string The name of the kafka topic to send the messages to.
messages IEnumerable The enumerable of messages that will be sent to the given topic.
acks System.Int16 The required level of acknowlegment from the kafka server. 0=none, 1=writen to leader, 2+=writen to replicas, -1=writen to all replicas.
timeout System.TimeSpan Interal kafka timeout to wait for the requested level of ack to occur before returning. Defaults to 1000ms.
codec MessageCodec The codec to apply to the message collection. Defaults to none.
partition int
return Task

SendMessageAsync() public method

public SendMessageAsync ( string topic, int partition ) : Task
topic string
partition int
return Task

Stop() public method

Stops the producer from accepting new messages, and optionally waits for in-flight messages to be sent before returning.
public Stop ( bool waitForRequestsToComplete = true, System.TimeSpan maxWait = null ) : void
waitForRequestsToComplete bool True to wait for in-flight requests to complete, false otherwise.
maxWait System.TimeSpan Maximum time to wait for in-flight requests to complete. Has no effect if waitForRequestsToComplete is false
return void