CloudWiki

Amazon Web Service (AWS)

Kinesis

Queue
Amazon Kinesis is a real-time data streaming service designed to process large scale data streams at any scale.
aws_kinesis_stream
Kinesis
attributes:
  • name - (Required) A name to identify the stream. This is unique to the AWS account and region the Stream is created in.
  • shard_count – (Optional) The number of shards that the stream will use. If the stream_mode is PROVISIONED, this field is required. Amazon has guidelines for specifying the Stream size that should be referenced when creating a Kinesis stream. See Amazon Kinesis Streams for more.
  • retention_period - (Optional) Length of time data records are accessible after they are added to the stream. The maximum value of a stream's retention period is 8760 hours. Minimum value is 24. Default is 24.
  • shard_level_metrics - (Optional) A list of shard-level CloudWatch metrics which can be enabled for the stream. See Monitoring with CloudWatch for more. Note that the value ALL should not be used; instead you should provide an explicit list of metrics you wish to enable.
  • enforce_consumer_deletion - (Optional) A boolean that indicates all registered consumers should be deregistered from the stream so that the stream can be destroyed without error. The default value is false.
  • encryption_type - (Optional) The encryption type to use. The only acceptable values are NONE or KMS. The default value is NONE.
  • kms_key_id - (Optional) The GUID for the customer-managed KMS key to use for encryption. You can also use a Kinesis-owned master key by specifying the alias alias/aws/kinesis.
  • stream_mode_details - (Optional) Indicates the capacity mode of the data stream. Detailed below.
  • tags - (Optional) A map of tags to assign to the resource. If configured with a provider default_tags configuration block present, tags with matching keys will overwrite those defined at the provider-level.

stream_mode_details Configuration Block

  • stream_mode - (Required) Specifies the capacity mode of the stream. Must be either PROVISIONED or ON_DEMAND.

Associating resources with a
Kinesis
Resources do not "belong" to a
Kinesis
Rather, one or more Security Groups are associated to a resource.
Create
Kinesis
via Terraform:
The following HCL creates a Kinesis stream resource
Syntax:

resource "aws_kinesis_stream" "test_stream" {
 name             = "terraform-kinesis-test"
 shard_count      = 1
 retention_period = 48

 shard_level_metrics = [
   "IncomingBytes",
   "OutgoingBytes",
 ]

 stream_mode_details {
   stream_mode = "PROVISIONED"
 }

 tags = {
   Environment = "test"
 }
}

Create
Kinesis
via CLI:
Parametres:

create-stream
--stream-name <value>
[--shard-count <value>]
[--stream-mode-details <value>]
[--cli-input-json | --cli-input-yaml]
[--generate-cli-skeleton <value>]
[--debug]
[--endpoint-url <value>]
[--no-verify-ssl]
[--no-paginate]
[--output <value>]
[--query <value>]
[--profile <value>]
[--region <value>]
[--version <value>]
[--color <value>]
[--no-sign-request]
[--ca-bundle <value>]
[--cli-read-timeout <value>]
[--cli-connect-timeout <value>]
[--cli-binary-format <value>]
[--no-cli-pager]
[--cli-auto-prompt]
[--no-cli-auto-prompt]

Example:

aws kinesis create-stream \
   --stream-name samplestream \
   --shard-count 3

Best Practices for
Kinesis

Categorized by Availability, Security & Compliance and Cost

Warning
Ensure Kinesis Data Stream encryption is enabled
Explore all the rules our platform covers
All Resources