C# (CSharp) Amazon.ElasticMapReduce.Model Namespace

Nested Namespaces

Amazon.ElasticMapReduce.Model.Internal

Сlasses

Name Description
AddInstanceGroupsRequest Container for the parameters to the AddInstanceGroups operation.

AddInstanceGroups adds an instance group to a running cluster.

AddInstanceGroupsResponse Returns information about the AddInstanceGroups response and response metadata.
AddInstanceGroupsResult

Output from an AddInstanceGroups call.

AddJobFlowStepsRequest Container for the parameters to the AddJobFlowSteps operation. AddJobFlowSteps adds new steps to a running job flow. A maximum of 256 steps are allowed in each job flow.

If your job flow is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop. For more information on how to do this, see Add More than 256 Steps to a Job Flow in the Amazon EMR Developer's Guide.

A step specifies the location of a JAR file stored either on the master node of the job flow or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step.

Amazon EMR executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all Hadoop jobs started while the step was running must have completed and run successfully.

You can only add steps to a job flow that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING.

AddJobFlowStepsResponse
AddJobFlowStepsResult

The output for the AddJobFlowSteps operation.

AddTagsRequest Container for the parameters to the AddTags operation.

Adds tags to an Amazon EMR resource. Tags make it easier to associate clusters in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tagging Amazon EMR Resources .

AddTagsResponse Returns information about the AddTags response and response metadata.
Application An application is any Amazon or third-party software that you can add to the cluster. This structure contains a list of strings that indicates the software to use with the cluster and accepts a user argument list. Amazon EMR accepts and forwards the argument list to the corresponding installation script as bootstrap action argument. For more information, see Launch a Job Flow on the MapR Distribution for Hadoop. Currently supported values are:
  • "mapr-m3" - launch the job flow using MapR M3 Edition.

  • "mapr-m5" - launch the job flow using MapR M5 Edition.

  • "mapr" with the user arguments specifying "--edition,m3" or "--edition,m5" - launch the job flow using MapR M3 or M5 Edition, respectively.

In Amazon EMR releases 4.0 and greater, the only accepted parameter is the application name. To pass arguments to applications, you supply a configuration for each application.

AutoScalingPolicy An automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster. An automatic scaling policy defines how an instance group dynamically adds and terminates EC2 instances in response to the value of a CloudWatch metric. See PutAutoScalingPolicy.
AutoScalingPolicyDescription An automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster. The automatic scaling policy defines how an instance group dynamically adds and terminates EC2 instances in response to the value of a CloudWatch metric. See PutAutoScalingPolicy.
AutoScalingPolicyStateChangeReason The reason for an AutoScalingPolicyStatus change.
AutoScalingPolicyStatus The status of an automatic scaling policy.
BootstrapActionConfig

Configuration of a bootstrap action.

BootstrapActionDetail

Reports the configuration of a bootstrap action in a job flow.

CancelStepsInfo
CancelStepsRequest Container for the parameters to the CancelSteps operation. Cancels a pending step or steps in a running cluster. Available only in Amazon EMR versions 4.8.0 and later, excluding version 5.0.0. A maximum of 256 steps are allowed in each CancelSteps request. CancelSteps is idempotent but asynchronous; it does not guarantee a step will be canceled, even if the request is successfully submitted. You can only cancel steps that are in a PENDING state.
CancelStepsResponse The output for the CancelSteps operation.
CloudWatchAlarmDefinition The definition of a CloudWatch metric alarm, which determines when an automatic scaling activity is triggered. When the defined alarm conditions are satisfied, scaling activity begins.
Cluster The detailed description of the cluster.
ClusterStateChangeReason

The reason that the cluster changed to its current state.

ClusterStatus

The detailed status of the cluster.

ClusterSummary The summary description of the cluster.
ClusterTimeline

Represents the timeline of the cluster's lifecycle.

Command

An entity describing an executable that runs on a cluster.

Configuration

Amazon EMR releases 4.x or later.

Specifies a hardware and software configuration of the EMR cluster. This includes configurations for applications and software bundled with Amazon EMR. The Configuration object is a JSON object which is defined by a classification and a set of properties. Configurations can be nested, so a configuration may have its own Configuration objects listed.

CreateSecurityConfigurationRequest Container for the parameters to the CreateSecurityConfiguration operation. Creates a security configuration, which is stored in the service and can be specified when a cluster is created.
CreateSecurityConfigurationResponse This is the response object from the CreateSecurityConfiguration operation.
DeleteSecurityConfigurationRequest Container for the parameters to the DeleteSecurityConfiguration operation. Deletes a security configuration.
DescribeClusterRequest Container for the parameters to the DescribeCluster operation. Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on. For information about the cluster steps, see ListSteps.
DescribeClusterResponse Returns information about the DescribeCluster response and response metadata.
DescribeClusterResult This output contains the description of the cluster.
DescribeJobFlowsRequest Container for the parameters to the DescribeJobFlows operation.

DescribeJobFlows returns a list of job flows that match all of the supplied parameters. The parameters can include a list of job flow IDs, job flow states, and restrictions on job flow creation date and time.

Regardless of supplied parameters, only job flows created within the last two months are returned.

If no parameters are supplied, then job flows matching either of the following criteria are returned:

  • Job flows created and completed in the last two weeks
  • Job flows created within the last two months that are in one of the following states: RUNNING , WAITING , SHUTTING_DOWN , STARTING

Amazon Elastic MapReduce can return a maximum of 512 job flow descriptions.

DescribeJobFlowsResponse Returns information about the DescribeJobFlows response and response metadata.
DescribeJobFlowsResult
DescribeSecurityConfigurationRequest Container for the parameters to the DescribeSecurityConfiguration operation. Provides the details of a security configuration by returning the configuration JSON.
DescribeSecurityConfigurationResponse This is the response object from the DescribeSecurityConfiguration operation.
DescribeStepRequest Container for the parameters to the DescribeStep operation.

Provides more detail about the cluster step.

DescribeStepResponse This output contains the description of the cluster step.
DescribeStepResult

This output contains the description of the cluster step.

EbsBlockDevice Configuration of requested EBS block device associated with the instance group.
EbsBlockDeviceConfig Configuration of requested EBS block device associated with the instance group with count of volumes that will be associated to every instance.
EbsConfiguration The Amazon EBS configuration of a cluster instance.
EbsVolume EBS block device that's attached to an EC2 instance.
Ec2InstanceAttributes Provides information about the EC2 instances in a cluster grouped by category. For example, key name, subnet ID, IAM instance profile, and so on.
FailureDetails The details of the step failure. The service attempts to detect the root cause for many common failures.
HadoopJarStepConfig
HadoopStepConfig

A cluster step consisting of a JAR file whose main function will be executed. The main function submits a job for Hadoop to execute and waits for the job to finish or fail.

Instance Represents an EC2 instance provisioned as part of cluster.
InstanceGroup This entity represents an instance group, which is a group of instances that have common purpose. For example, CORE instance group is used for HDFS.
InstanceGroupConfig

Configuration defining a new instance group.

InstanceGroupDetail

Detailed information about an instance group.

InstanceGroupModifyConfig Modify an instance group size.
InstanceGroupStateChangeReason

The status change reason details for the instance group.

InstanceGroupStatus

The details of the instance group status.

InstanceGroupTimeline

The timeline of the instance group lifecycle.

InstanceResizePolicy Custom policy for requesting termination protection or termination of specific instances when shrinking an instance group.
InstanceStateChangeReason

The details of the status change reason for the instance.

InstanceStatus

The instance status details.

InstanceTimeline

The timeline of the instance lifecycle.

InternalServerErrorException
InternalServerException
InvalidRequestException
JobFlowDetail
JobFlowExecutionStatusDetail

Describes the status of the job flow.

JobFlowInstancesConfig
JobFlowInstancesDetail
KeyValue

A key value pair.

ListBootstrapActionsRequest Container for the parameters to the ListBootstrapActions operation. Provides information about the bootstrap actions associated with a cluster.
ListBootstrapActionsResponse Returns information about the ListBootstrapActions response and response metadata.
ListBootstrapActionsResult

This output contains the bootstrap actions detail.

ListClustersRequest Container for the parameters to the ListClusters operation. Provides the status of all clusters visible to this AWS account. Allows you to filter the list of clusters based on certain criteria; for example, filtering by cluster creation date and time or by status. This call returns a maximum of 50 clusters per call, but returns a marker to track the paging of the cluster list across multiple ListClusters calls.
ListClustersResponse Returns information about the ListClusters response and response metadata.
ListInstanceGroupsRequest Container for the parameters to the ListInstanceGroups operation.

Provides all available details about the instance groups in a cluster.

ListInstanceGroupsResponse Returns information about the ListInstanceGroups response and response metadata.
ListInstanceGroupsResult This input determines which instance groups to retrieve.
ListInstancesRequest Container for the parameters to the ListInstances operation. Provides information about the cluster instances that Amazon EMR provisions on behalf of a user when it creates the cluster. For example, this operation indicates when the EC2 instances reach the Ready state, when instances become available to Amazon EMR to use for jobs, and the IP addresses for cluster instances, etc.
ListInstancesResponse This output contains the list of instances.
ListInstancesResult This output contains the list of instances.
ListSecurityConfigurationsRequest Container for the parameters to the ListSecurityConfigurations operation. Lists all the security configurations visible to this account, providing their creation dates and times, and their names. This call returns a maximum of 50 clusters per call, but returns a marker to track the paging of the cluster list across multiple ListSecurityConfigurations calls.
ListSecurityConfigurationsResponse This is the response object from the ListSecurityConfigurations operation.
ListStepsRequest Container for the parameters to the ListSteps operation.

Provides a list of steps for the cluster.

ListStepsResponse This output contains the list of steps returned in reverse order. This means that the last step is the first element in the list.
ListStepsResult This output contains the list of steps.
MetricDimension A CloudWatch dimension, which is specified using a Key (known as a Name in CloudWatch), Value pair. By default, Amazon EMR uses one dimension whose Key is JobFlowID and Value is a variable representing the cluster ID, which is ${emr:cluster_id}. This enables the rule to bootstrap when the cluster ID becomes available, and also enables a single automatic scaling policy to be reused for multiple clusters and instance groups.
ModifyInstanceGroupsRequest Container for the parameters to the ModifyInstanceGroups operation.

ModifyInstanceGroups modifies the number of nodes and configuration settings of an instance group. The input parameters include the new target instance count for the group and the instance group ID. The call will either succeed or fail atomically.

PlacementType
PutAutoScalingPolicyRequest Container for the parameters to the PutAutoScalingPolicy operation. Creates or updates an automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster. The automatic scaling policy defines how an instance group dynamically adds and terminates EC2 instances in response to the value of a CloudWatch metric.
PutAutoScalingPolicyResponse This is the response object from the PutAutoScalingPolicy operation.
RemoveAutoScalingPolicyRequest Container for the parameters to the RemoveAutoScalingPolicy operation. Removes an automatic scaling policy from a specified instance group within an EMR cluster.
RemoveTagsRequest Container for the parameters to the RemoveTags operation. Removes tags from an Amazon EMR resource. Tags make it easier to associate clusters in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tagging Amazon EMR Resources.

The following example removes the stack tag with value Prod from a cluster:

RemoveTagsResponse Returns information about the RemoveTags response and response metadata.
RunJobFlowRequest Container for the parameters to the RunJobFlow operation.

RunJobFlow creates and starts running a new job flow. The job flow will run the steps specified. Once the job flow completes, the cluster is stopped and the HDFS partition is lost. To prevent loss of data, configure the last step of the job flow to store results in Amazon S3. If the JobFlowInstancesDetail KeepJobFlowAliveWhenNoSteps parameter is set to TRUE , the job flow will transition to the WAITING state rather than shutting down once the steps have completed.

For additional protection, you can set the JobFlowInstancesDetail TerminationProtected parameter to TRUE to lock the job flow and prevent it from being terminated by API call, user intervention, or in the event of a job flow error.

A maximum of 256 steps are allowed in each job flow.

If your job flow is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop. For more information on how to do this, go to Add More than 256 Steps to a Job Flow in the Amazon Elastic MapReduce Developer's Guide .

For long running job flows, we recommend that you periodically store your results.

RunJobFlowResponse Returns information about the RunJobFlowResult response and response metadata.
RunJobFlowResult The result of the RunJobFlow operation.
ScalingAction The type of adjustment the automatic scaling activity makes when triggered, and the periodicity of the adjustment.
ScalingConstraints The upper and lower EC2 instance limits for an automatic scaling policy. Automatic scaling activities triggered by automatic scaling rules will not cause an instance group to grow above or below these limits.
ScalingRule A scale-in or scale-out rule that defines scaling activity, including the CloudWatch metric alarm that triggers activity, how EC2 instances are added or removed, and the periodicity of adjustments. The automatic scaling policy for an instance group can comprise one or more automatic scaling rules.
ScalingTrigger The conditions that trigger an automatic scaling activity.
ScriptBootstrapActionConfig

Configuration of the script to run during a bootstrap action.

SecurityConfigurationSummary The creation date and time, and name, of a security configuration.
SetTerminationProtectionRequest Container for the parameters to the SetTerminationProtection operation. SetTerminationProtection locks a job flow so the EC2 instances in the cluster cannot be terminated by user intervention, an API call, or in the event of a job-flow error. The cluster still terminates upon successful completion of the job flow. Calling SetTerminationProtection on a job flow is analogous to calling the Amazon EC2 DisableAPITermination API on all of the EC2 instances in a cluster.

SetTerminationProtection is used to prevent accidental termination of a job flow and to ensure that in the event of an error, the instances will persist so you can recover any data stored in their ephemeral instance storage.

To terminate a job flow that has been locked by setting SetTerminationProtection to true, you must first unlock the job flow by a subsequent call to SetTerminationProtection in which you set the value to false.

For more information, seeProtecting a Job Flow from Termination in the Amazon EMR Guide.

SetVisibleToAllUsersRequest Container for the parameters to the SetVisibleToAllUsers operation.

Sets whether all AWS Identity and Access Management (IAM) users under your account can access the specifed job flows. This action works on running job flows. You can also set the visibility of a job flow when you launch it using the VisibleToAllUsers parameter of RunJobFlow. The SetVisibleToAllUsers action can be called only by an IAM user who created the job flow or the AWS account that owns the job flow.

ShrinkPolicy Policy for customizing shrink operations. Allows configuration of decommissioning timeout and targeted instance shrinking.
SimpleScalingPolicyConfiguration An automatic scaling configuration, which describes how the policy adds or removes instances, the cooldown period, and the number of EC2 instances that will be added each time the CloudWatch metric alarm condition is satisfied.
Step This represents a step in a cluster.
StepConfig

Specification of a job flow step.

StepDetail

Combines the execution state and configuration of a step.

StepExecutionStatusDetail
StepFactory This class provides helper methods for creating common Elastic MapReduce step types. To use StepFactory, you should construct it with the appropriate bucket for your region. The official bucket format is "<region>.elasticmapreduce", so us-east-1 would use the bucket "us-east-1.elasticmapreduce".
StepFactory.HiveVersion The available Hive versions. For reference: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_SupportedHiveVersions.html
StepStateChangeReason

The details of the step state change reason.

StepStatus The execution status details of the cluster step.
StepSummary

The summary of the cluster step.

StepTimeline

The timeline of the cluster step lifecycle.

SupportedProductConfig The list of supported product configurations which allow user-supplied arguments. EMR accepts these arguments and forwards them to the corresponding installation script as bootstrap action arguments.
Tag

A key/value pair that contains user-defined metadata that you can associate with an Amazon EMR resource. Tags make it easier to associate clusters in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tagging Amazon EMR Resources .

TerminateJobFlowsRequest
VolumeSpecification EBS volume specifications such as volume type, IOPS, and size (GiB) that will be requested for the EBS volume attached to an EC2 instance in the cluster.