Docs Home Pulumi IaC Pulumi ESC Pulumi Cloud Packages Tutorials
  1. Packages
  2. Google Cloud (GCP) Classic
  3. API Docs
  4. dataflow
  5. FlexTemplateJob
Google Cloud v8.21.0 published on Wednesday, Mar 5, 2025 by Pulumi
pulumi/pulumi-gcp

gcp.dataflow.FlexTemplateJob

Explore with Pulumi AI

Google Cloud v8.21.0 published on Wednesday, Mar 5, 2025 by Pulumi
pulumi/pulumi-gcp

Example Usage

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";

const bigDataJob = new gcp.dataflow.FlexTemplateJob("big_data_job", {
    name: "dataflow-flextemplates-job",
    containerSpecGcsPath: "gs://my-bucket/templates/template.json",
    parameters: {
        inputSubscription: "messages",
    },
});
Copy
import pulumi
import pulumi_gcp as gcp

big_data_job = gcp.dataflow.FlexTemplateJob("big_data_job",
    name="dataflow-flextemplates-job",
    container_spec_gcs_path="gs://my-bucket/templates/template.json",
    parameters={
        "inputSubscription": "messages",
    })
Copy
package main

import (
	"github.com/pulumi/pulumi-gcp/sdk/v8/go/gcp/dataflow"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		_, err := dataflow.NewFlexTemplateJob(ctx, "big_data_job", &dataflow.FlexTemplateJobArgs{
			Name:                 pulumi.String("dataflow-flextemplates-job"),
			ContainerSpecGcsPath: pulumi.String("gs://my-bucket/templates/template.json"),
			Parameters: pulumi.StringMap{
				"inputSubscription": pulumi.String("messages"),
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}
Copy
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;

return await Deployment.RunAsync(() => 
{
    var bigDataJob = new Gcp.Dataflow.FlexTemplateJob("big_data_job", new()
    {
        Name = "dataflow-flextemplates-job",
        ContainerSpecGcsPath = "gs://my-bucket/templates/template.json",
        Parameters = 
        {
            { "inputSubscription", "messages" },
        },
    });

});
Copy
package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.dataflow.FlexTemplateJob;
import com.pulumi.gcp.dataflow.FlexTemplateJobArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        var bigDataJob = new FlexTemplateJob("bigDataJob", FlexTemplateJobArgs.builder()
            .name("dataflow-flextemplates-job")
            .containerSpecGcsPath("gs://my-bucket/templates/template.json")
            .parameters(Map.of("inputSubscription", "messages"))
            .build());

    }
}
Copy
resources:
  bigDataJob:
    type: gcp:dataflow:FlexTemplateJob
    name: big_data_job
    properties:
      name: dataflow-flextemplates-job
      containerSpecGcsPath: gs://my-bucket/templates/template.json
      parameters:
        inputSubscription: messages
Copy

Note on “destroy” / “apply”

There are many types of Dataflow jobs. Some Dataflow jobs run constantly, getting new data from (e.g.) a GCS bucket, and outputting data continuously. Some jobs process a set amount of data then terminate. All jobs can fail while running due to programming errors or other issues. In this way, Dataflow jobs are different from most other provider / Google resources.

The Dataflow resource is considered ’existing’ while it is in a nonterminal state. If it reaches a terminal state (e.g. ‘FAILED’, ‘COMPLETE’, ‘CANCELLED’), it will be recreated on the next ‘apply’. This is as expected for jobs which run continuously, but may surprise users who use this resource for other kinds of Dataflow jobs.

A Dataflow job which is ‘destroyed’ may be “cancelled” or “drained”. If “cancelled”, the job terminates - any data written remains where it is, but no new data will be processed. If “drained”, no new data will enter the pipeline, but any data currently in the pipeline will finish being processed. The default is “cancelled”, but if a user sets on_delete to "drain" in the configuration, you may experience a long wait for your pulumi destroy to complete.

You can potentially short-circuit the wait by setting skip_wait_on_job_termination to true, but beware that unless you take active steps to ensure that the job name parameter changes between instances, the name will conflict and the launch of the new job will fail. One way to do this is with a random_id resource, for example:

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import * as random from "@pulumi/random";

const config = new pulumi.Config();
const bigDataJobSubscriptionId = config.get("bigDataJobSubscriptionId") || "projects/myproject/subscriptions/messages";
const bigDataJobNameSuffix = new random.RandomId("big_data_job_name_suffix", {
    byteLength: 4,
    keepers: {
        region: region,
        subscription_id: bigDataJobSubscriptionId,
    },
});
const bigDataJob = new gcp.dataflow.FlexTemplateJob("big_data_job", {
    name: pulumi.interpolate`dataflow-flextemplates-job-${bigDataJobNameSuffix.dec}`,
    region: region,
    containerSpecGcsPath: "gs://my-bucket/templates/template.json",
    skipWaitOnJobTermination: true,
    parameters: {
        inputSubscription: bigDataJobSubscriptionId,
    },
});
Copy
import pulumi
import pulumi_gcp as gcp
import pulumi_random as random

config = pulumi.Config()
big_data_job_subscription_id = config.get("bigDataJobSubscriptionId")
if big_data_job_subscription_id is None:
    big_data_job_subscription_id = "projects/myproject/subscriptions/messages"
big_data_job_name_suffix = random.RandomId("big_data_job_name_suffix",
    byte_length=4,
    keepers={
        "region": region,
        "subscription_id": big_data_job_subscription_id,
    })
big_data_job = gcp.dataflow.FlexTemplateJob("big_data_job",
    name=big_data_job_name_suffix.dec.apply(lambda dec: f"dataflow-flextemplates-job-{dec}"),
    region=region,
    container_spec_gcs_path="gs://my-bucket/templates/template.json",
    skip_wait_on_job_termination=True,
    parameters={
        "inputSubscription": big_data_job_subscription_id,
    })
Copy
package main

import (
	"fmt"

	"github.com/pulumi/pulumi-gcp/sdk/v8/go/gcp/dataflow"
	"github.com/pulumi/pulumi-random/sdk/v4/go/random"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		cfg := config.New(ctx, "")
		bigDataJobSubscriptionId := "projects/myproject/subscriptions/messages"
		if param := cfg.Get("bigDataJobSubscriptionId"); param != "" {
			bigDataJobSubscriptionId = param
		}
		bigDataJobNameSuffix, err := random.NewRandomId(ctx, "big_data_job_name_suffix", &random.RandomIdArgs{
			ByteLength: pulumi.Int(4),
			Keepers: pulumi.StringMap{
				"region":          pulumi.Any(region),
				"subscription_id": pulumi.String(bigDataJobSubscriptionId),
			},
		})
		if err != nil {
			return err
		}
		_, err = dataflow.NewFlexTemplateJob(ctx, "big_data_job", &dataflow.FlexTemplateJobArgs{
			Name: bigDataJobNameSuffix.Dec.ApplyT(func(dec string) (string, error) {
				return fmt.Sprintf("dataflow-flextemplates-job-%v", dec), nil
			}).(pulumi.StringOutput),
			Region:                   pulumi.Any(region),
			ContainerSpecGcsPath:     pulumi.String("gs://my-bucket/templates/template.json"),
			SkipWaitOnJobTermination: pulumi.Bool(true),
			Parameters: pulumi.StringMap{
				"inputSubscription": pulumi.String(bigDataJobSubscriptionId),
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}
Copy
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
using Random = Pulumi.Random;

return await Deployment.RunAsync(() => 
{
    var config = new Config();
    var bigDataJobSubscriptionId = config.Get("bigDataJobSubscriptionId") ?? "projects/myproject/subscriptions/messages";
    var bigDataJobNameSuffix = new Random.RandomId("big_data_job_name_suffix", new()
    {
        ByteLength = 4,
        Keepers = 
        {
            { "region", region },
            { "subscription_id", bigDataJobSubscriptionId },
        },
    });

    var bigDataJob = new Gcp.Dataflow.FlexTemplateJob("big_data_job", new()
    {
        Name = bigDataJobNameSuffix.Dec.Apply(dec => $"dataflow-flextemplates-job-{dec}"),
        Region = region,
        ContainerSpecGcsPath = "gs://my-bucket/templates/template.json",
        SkipWaitOnJobTermination = true,
        Parameters = 
        {
            { "inputSubscription", bigDataJobSubscriptionId },
        },
    });

});
Copy
package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.random.RandomId;
import com.pulumi.random.RandomIdArgs;
import com.pulumi.gcp.dataflow.FlexTemplateJob;
import com.pulumi.gcp.dataflow.FlexTemplateJobArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        final var config = ctx.config();
        final var bigDataJobSubscriptionId = config.get("bigDataJobSubscriptionId").orElse("projects/myproject/subscriptions/messages");
        var bigDataJobNameSuffix = new RandomId("bigDataJobNameSuffix", RandomIdArgs.builder()
            .byteLength(4)
            .keepers(Map.ofEntries(
                Map.entry("region", region),
                Map.entry("subscription_id", bigDataJobSubscriptionId)
            ))
            .build());

        var bigDataJob = new FlexTemplateJob("bigDataJob", FlexTemplateJobArgs.builder()
            .name(bigDataJobNameSuffix.dec().applyValue(dec -> String.format("dataflow-flextemplates-job-%s", dec)))
            .region(region)
            .containerSpecGcsPath("gs://my-bucket/templates/template.json")
            .skipWaitOnJobTermination(true)
            .parameters(Map.of("inputSubscription", bigDataJobSubscriptionId))
            .build());

    }
}
Copy
configuration:
  bigDataJobSubscriptionId:
    type: string
    default: projects/myproject/subscriptions/messages
resources:
  bigDataJobNameSuffix:
    type: random:RandomId
    name: big_data_job_name_suffix
    properties:
      byteLength: 4
      keepers:
        region: ${region}
        subscription_id: ${bigDataJobSubscriptionId}
  bigDataJob:
    type: gcp:dataflow:FlexTemplateJob
    name: big_data_job
    properties:
      name: dataflow-flextemplates-job-${bigDataJobNameSuffix.dec}
      region: ${region}
      containerSpecGcsPath: gs://my-bucket/templates/template.json
      skipWaitOnJobTermination: true
      parameters:
        inputSubscription: ${bigDataJobSubscriptionId}
Copy

Create FlexTemplateJob Resource

Resources are created with functions called constructors. To learn more about declaring and configuring resources, see Resources.

Constructor syntax

new FlexTemplateJob(name: string, args: FlexTemplateJobArgs, opts?: CustomResourceOptions);
@overload
def FlexTemplateJob(resource_name: str,
                    args: FlexTemplateJobArgs,
                    opts: Optional[ResourceOptions] = None)

@overload
def FlexTemplateJob(resource_name: str,
                    opts: Optional[ResourceOptions] = None,
                    container_spec_gcs_path: Optional[str] = None,
                    num_workers: Optional[int] = None,
                    transform_name_mapping: Optional[Mapping[str, str]] = None,
                    network: Optional[str] = None,
                    ip_configuration: Optional[str] = None,
                    kms_key_name: Optional[str] = None,
                    labels: Optional[Mapping[str, str]] = None,
                    launcher_machine_type: Optional[str] = None,
                    machine_type: Optional[str] = None,
                    max_workers: Optional[int] = None,
                    on_delete: Optional[str] = None,
                    enable_streaming_engine: Optional[bool] = None,
                    autoscaling_algorithm: Optional[str] = None,
                    name: Optional[str] = None,
                    parameters: Optional[Mapping[str, str]] = None,
                    project: Optional[str] = None,
                    region: Optional[str] = None,
                    sdk_container_image: Optional[str] = None,
                    service_account_email: Optional[str] = None,
                    skip_wait_on_job_termination: Optional[bool] = None,
                    staging_location: Optional[str] = None,
                    subnetwork: Optional[str] = None,
                    temp_location: Optional[str] = None,
                    additional_experiments: Optional[Sequence[str]] = None)
func NewFlexTemplateJob(ctx *Context, name string, args FlexTemplateJobArgs, opts ...ResourceOption) (*FlexTemplateJob, error)
public FlexTemplateJob(string name, FlexTemplateJobArgs args, CustomResourceOptions? opts = null)
public FlexTemplateJob(String name, FlexTemplateJobArgs args)
public FlexTemplateJob(String name, FlexTemplateJobArgs args, CustomResourceOptions options)

type: gcp:dataflow:FlexTemplateJob
properties: # The arguments to resource properties.
options: # Bag of options to control resource's behavior.

Parameters

name This property is required. string
The unique name of the resource.
args This property is required. FlexTemplateJobArgs
The arguments to resource properties.
opts CustomResourceOptions
Bag of options to control resource's behavior.
resource_name This property is required. str
The unique name of the resource.
args This property is required. FlexTemplateJobArgs
The arguments to resource properties.
opts ResourceOptions
Bag of options to control resource's behavior.
ctx Context
Context object for the current deployment.
name This property is required. string
The unique name of the resource.
args This property is required. FlexTemplateJobArgs
The arguments to resource properties.
opts ResourceOption
Bag of options to control resource's behavior.
name This property is required. string
The unique name of the resource.
args This property is required. FlexTemplateJobArgs
The arguments to resource properties.
opts CustomResourceOptions
Bag of options to control resource's behavior.
name This property is required. String
The unique name of the resource.
args This property is required. FlexTemplateJobArgs
The arguments to resource properties.
options CustomResourceOptions
Bag of options to control resource's behavior.

Constructor example

The following reference example uses placeholder values for all input properties.

var flexTemplateJobResource = new Gcp.Dataflow.FlexTemplateJob("flexTemplateJobResource", new()
{
    ContainerSpecGcsPath = "string",
    NumWorkers = 0,
    TransformNameMapping = 
    {
        { "string", "string" },
    },
    Network = "string",
    IpConfiguration = "string",
    KmsKeyName = "string",
    Labels = 
    {
        { "string", "string" },
    },
    LauncherMachineType = "string",
    MachineType = "string",
    MaxWorkers = 0,
    OnDelete = "string",
    EnableStreamingEngine = false,
    AutoscalingAlgorithm = "string",
    Name = "string",
    Parameters = 
    {
        { "string", "string" },
    },
    Project = "string",
    Region = "string",
    SdkContainerImage = "string",
    ServiceAccountEmail = "string",
    SkipWaitOnJobTermination = false,
    StagingLocation = "string",
    Subnetwork = "string",
    TempLocation = "string",
    AdditionalExperiments = new[]
    {
        "string",
    },
});
Copy
example, err := dataflow.NewFlexTemplateJob(ctx, "flexTemplateJobResource", &dataflow.FlexTemplateJobArgs{
	ContainerSpecGcsPath: pulumi.String("string"),
	NumWorkers:           pulumi.Int(0),
	TransformNameMapping: pulumi.StringMap{
		"string": pulumi.String("string"),
	},
	Network:         pulumi.String("string"),
	IpConfiguration: pulumi.String("string"),
	KmsKeyName:      pulumi.String("string"),
	Labels: pulumi.StringMap{
		"string": pulumi.String("string"),
	},
	LauncherMachineType:   pulumi.String("string"),
	MachineType:           pulumi.String("string"),
	MaxWorkers:            pulumi.Int(0),
	OnDelete:              pulumi.String("string"),
	EnableStreamingEngine: pulumi.Bool(false),
	AutoscalingAlgorithm:  pulumi.String("string"),
	Name:                  pulumi.String("string"),
	Parameters: pulumi.StringMap{
		"string": pulumi.String("string"),
	},
	Project:                  pulumi.String("string"),
	Region:                   pulumi.String("string"),
	SdkContainerImage:        pulumi.String("string"),
	ServiceAccountEmail:      pulumi.String("string"),
	SkipWaitOnJobTermination: pulumi.Bool(false),
	StagingLocation:          pulumi.String("string"),
	Subnetwork:               pulumi.String("string"),
	TempLocation:             pulumi.String("string"),
	AdditionalExperiments: pulumi.StringArray{
		pulumi.String("string"),
	},
})
Copy
var flexTemplateJobResource = new FlexTemplateJob("flexTemplateJobResource", FlexTemplateJobArgs.builder()
    .containerSpecGcsPath("string")
    .numWorkers(0)
    .transformNameMapping(Map.of("string", "string"))
    .network("string")
    .ipConfiguration("string")
    .kmsKeyName("string")
    .labels(Map.of("string", "string"))
    .launcherMachineType("string")
    .machineType("string")
    .maxWorkers(0)
    .onDelete("string")
    .enableStreamingEngine(false)
    .autoscalingAlgorithm("string")
    .name("string")
    .parameters(Map.of("string", "string"))
    .project("string")
    .region("string")
    .sdkContainerImage("string")
    .serviceAccountEmail("string")
    .skipWaitOnJobTermination(false)
    .stagingLocation("string")
    .subnetwork("string")
    .tempLocation("string")
    .additionalExperiments("string")
    .build());
Copy
flex_template_job_resource = gcp.dataflow.FlexTemplateJob("flexTemplateJobResource",
    container_spec_gcs_path="string",
    num_workers=0,
    transform_name_mapping={
        "string": "string",
    },
    network="string",
    ip_configuration="string",
    kms_key_name="string",
    labels={
        "string": "string",
    },
    launcher_machine_type="string",
    machine_type="string",
    max_workers=0,
    on_delete="string",
    enable_streaming_engine=False,
    autoscaling_algorithm="string",
    name="string",
    parameters={
        "string": "string",
    },
    project="string",
    region="string",
    sdk_container_image="string",
    service_account_email="string",
    skip_wait_on_job_termination=False,
    staging_location="string",
    subnetwork="string",
    temp_location="string",
    additional_experiments=["string"])
Copy
const flexTemplateJobResource = new gcp.dataflow.FlexTemplateJob("flexTemplateJobResource", {
    containerSpecGcsPath: "string",
    numWorkers: 0,
    transformNameMapping: {
        string: "string",
    },
    network: "string",
    ipConfiguration: "string",
    kmsKeyName: "string",
    labels: {
        string: "string",
    },
    launcherMachineType: "string",
    machineType: "string",
    maxWorkers: 0,
    onDelete: "string",
    enableStreamingEngine: false,
    autoscalingAlgorithm: "string",
    name: "string",
    parameters: {
        string: "string",
    },
    project: "string",
    region: "string",
    sdkContainerImage: "string",
    serviceAccountEmail: "string",
    skipWaitOnJobTermination: false,
    stagingLocation: "string",
    subnetwork: "string",
    tempLocation: "string",
    additionalExperiments: ["string"],
});
Copy
type: gcp:dataflow:FlexTemplateJob
properties:
    additionalExperiments:
        - string
    autoscalingAlgorithm: string
    containerSpecGcsPath: string
    enableStreamingEngine: false
    ipConfiguration: string
    kmsKeyName: string
    labels:
        string: string
    launcherMachineType: string
    machineType: string
    maxWorkers: 0
    name: string
    network: string
    numWorkers: 0
    onDelete: string
    parameters:
        string: string
    project: string
    region: string
    sdkContainerImage: string
    serviceAccountEmail: string
    skipWaitOnJobTermination: false
    stagingLocation: string
    subnetwork: string
    tempLocation: string
    transformNameMapping:
        string: string
Copy

FlexTemplateJob Resource Properties

To learn more about resource properties and how to use them, see Inputs and Outputs in the Architecture and Concepts docs.

Inputs

In Python, inputs that are objects can be passed either as argument classes or as dictionary literals.

The FlexTemplateJob resource accepts the following input properties:

ContainerSpecGcsPath This property is required. string
The GCS path to the Dataflow job Flex Template.

AdditionalExperiments List<string>
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
AutoscalingAlgorithm string
The algorithm to use for autoscaling.
EnableStreamingEngine Changes to this property will trigger replacement. bool
Immutable. Indicates if the job should use the streaming engine feature.
IpConfiguration string
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
KmsKeyName string
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
Labels Dictionary<string, string>
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
LauncherMachineType string
The machine type to use for launching the job. The default is n1-standard-1.
MachineType string
The machine type to use for the job.
MaxWorkers Changes to this property will trigger replacement. int
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
Name Changes to this property will trigger replacement. string
Immutable. A unique name for the resource, required by Dataflow.
Network string
The network to which VMs will be assigned. If it is not provided, "default" will be used.
NumWorkers Changes to this property will trigger replacement. int
Immutable. The initial number of Google Compute Engine instances for the job.
OnDelete string
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
Parameters Dictionary<string, string>
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
Project Changes to this property will trigger replacement. string
The project in which the resource belongs. If it is not provided, the provider project is used.
Region Changes to this property will trigger replacement. string
Immutable. The region in which the created job should run.
SdkContainerImage string
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
ServiceAccountEmail string
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
SkipWaitOnJobTermination bool
StagingLocation string
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
Subnetwork string
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
TempLocation string
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
TransformNameMapping Dictionary<string, string>
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
ContainerSpecGcsPath This property is required. string
The GCS path to the Dataflow job Flex Template.

AdditionalExperiments []string
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
AutoscalingAlgorithm string
The algorithm to use for autoscaling.
EnableStreamingEngine Changes to this property will trigger replacement. bool
Immutable. Indicates if the job should use the streaming engine feature.
IpConfiguration string
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
KmsKeyName string
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
Labels map[string]string
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
LauncherMachineType string
The machine type to use for launching the job. The default is n1-standard-1.
MachineType string
The machine type to use for the job.
MaxWorkers Changes to this property will trigger replacement. int
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
Name Changes to this property will trigger replacement. string
Immutable. A unique name for the resource, required by Dataflow.
Network string
The network to which VMs will be assigned. If it is not provided, "default" will be used.
NumWorkers Changes to this property will trigger replacement. int
Immutable. The initial number of Google Compute Engine instances for the job.
OnDelete string
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
Parameters map[string]string
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
Project Changes to this property will trigger replacement. string
The project in which the resource belongs. If it is not provided, the provider project is used.
Region Changes to this property will trigger replacement. string
Immutable. The region in which the created job should run.
SdkContainerImage string
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
ServiceAccountEmail string
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
SkipWaitOnJobTermination bool
StagingLocation string
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
Subnetwork string
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
TempLocation string
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
TransformNameMapping map[string]string
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
containerSpecGcsPath This property is required. String
The GCS path to the Dataflow job Flex Template.

additionalExperiments List<String>
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
autoscalingAlgorithm String
The algorithm to use for autoscaling.
enableStreamingEngine Changes to this property will trigger replacement. Boolean
Immutable. Indicates if the job should use the streaming engine feature.
ipConfiguration String
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
kmsKeyName String
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
labels Map<String,String>
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
launcherMachineType String
The machine type to use for launching the job. The default is n1-standard-1.
machineType String
The machine type to use for the job.
maxWorkers Changes to this property will trigger replacement. Integer
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
name Changes to this property will trigger replacement. String
Immutable. A unique name for the resource, required by Dataflow.
network String
The network to which VMs will be assigned. If it is not provided, "default" will be used.
numWorkers Changes to this property will trigger replacement. Integer
Immutable. The initial number of Google Compute Engine instances for the job.
onDelete String
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
parameters Map<String,String>
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
project Changes to this property will trigger replacement. String
The project in which the resource belongs. If it is not provided, the provider project is used.
region Changes to this property will trigger replacement. String
Immutable. The region in which the created job should run.
sdkContainerImage String
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
serviceAccountEmail String
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
skipWaitOnJobTermination Boolean
stagingLocation String
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
subnetwork String
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
tempLocation String
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
transformNameMapping Map<String,String>
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
containerSpecGcsPath This property is required. string
The GCS path to the Dataflow job Flex Template.

additionalExperiments string[]
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
autoscalingAlgorithm string
The algorithm to use for autoscaling.
enableStreamingEngine Changes to this property will trigger replacement. boolean
Immutable. Indicates if the job should use the streaming engine feature.
ipConfiguration string
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
kmsKeyName string
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
labels {[key: string]: string}
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
launcherMachineType string
The machine type to use for launching the job. The default is n1-standard-1.
machineType string
The machine type to use for the job.
maxWorkers Changes to this property will trigger replacement. number
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
name Changes to this property will trigger replacement. string
Immutable. A unique name for the resource, required by Dataflow.
network string
The network to which VMs will be assigned. If it is not provided, "default" will be used.
numWorkers Changes to this property will trigger replacement. number
Immutable. The initial number of Google Compute Engine instances for the job.
onDelete string
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
parameters {[key: string]: string}
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
project Changes to this property will trigger replacement. string
The project in which the resource belongs. If it is not provided, the provider project is used.
region Changes to this property will trigger replacement. string
Immutable. The region in which the created job should run.
sdkContainerImage string
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
serviceAccountEmail string
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
skipWaitOnJobTermination boolean
stagingLocation string
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
subnetwork string
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
tempLocation string
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
transformNameMapping {[key: string]: string}
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
container_spec_gcs_path This property is required. str
The GCS path to the Dataflow job Flex Template.

additional_experiments Sequence[str]
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
autoscaling_algorithm str
The algorithm to use for autoscaling.
enable_streaming_engine Changes to this property will trigger replacement. bool
Immutable. Indicates if the job should use the streaming engine feature.
ip_configuration str
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
kms_key_name str
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
labels Mapping[str, str]
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
launcher_machine_type str
The machine type to use for launching the job. The default is n1-standard-1.
machine_type str
The machine type to use for the job.
max_workers Changes to this property will trigger replacement. int
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
name Changes to this property will trigger replacement. str
Immutable. A unique name for the resource, required by Dataflow.
network str
The network to which VMs will be assigned. If it is not provided, "default" will be used.
num_workers Changes to this property will trigger replacement. int
Immutable. The initial number of Google Compute Engine instances for the job.
on_delete str
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
parameters Mapping[str, str]
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
project Changes to this property will trigger replacement. str
The project in which the resource belongs. If it is not provided, the provider project is used.
region Changes to this property will trigger replacement. str
Immutable. The region in which the created job should run.
sdk_container_image str
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
service_account_email str
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
skip_wait_on_job_termination bool
staging_location str
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
subnetwork str
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
temp_location str
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
transform_name_mapping Mapping[str, str]
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
containerSpecGcsPath This property is required. String
The GCS path to the Dataflow job Flex Template.

additionalExperiments List<String>
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
autoscalingAlgorithm String
The algorithm to use for autoscaling.
enableStreamingEngine Changes to this property will trigger replacement. Boolean
Immutable. Indicates if the job should use the streaming engine feature.
ipConfiguration String
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
kmsKeyName String
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
labels Map<String>
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
launcherMachineType String
The machine type to use for launching the job. The default is n1-standard-1.
machineType String
The machine type to use for the job.
maxWorkers Changes to this property will trigger replacement. Number
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
name Changes to this property will trigger replacement. String
Immutable. A unique name for the resource, required by Dataflow.
network String
The network to which VMs will be assigned. If it is not provided, "default" will be used.
numWorkers Changes to this property will trigger replacement. Number
Immutable. The initial number of Google Compute Engine instances for the job.
onDelete String
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
parameters Map<String>
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
project Changes to this property will trigger replacement. String
The project in which the resource belongs. If it is not provided, the provider project is used.
region Changes to this property will trigger replacement. String
Immutable. The region in which the created job should run.
sdkContainerImage String
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
serviceAccountEmail String
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
skipWaitOnJobTermination Boolean
stagingLocation String
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
subnetwork String
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
tempLocation String
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
transformNameMapping Map<String>
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.

Outputs

All input properties are implicitly available as output properties. Additionally, the FlexTemplateJob resource produces the following output properties:

EffectiveLabels Dictionary<string, string>
Id string
The provider-assigned unique ID for this managed resource.
JobId string
The unique ID of this job.
PulumiLabels Dictionary<string, string>
The combination of labels configured directly on the resource and default labels configured on the provider.
State string
The current state of the resource, selected from the JobState enum
Type string
The type of this job, selected from the JobType enum.
EffectiveLabels map[string]string
Id string
The provider-assigned unique ID for this managed resource.
JobId string
The unique ID of this job.
PulumiLabels map[string]string
The combination of labels configured directly on the resource and default labels configured on the provider.
State string
The current state of the resource, selected from the JobState enum
Type string
The type of this job, selected from the JobType enum.
effectiveLabels Map<String,String>
id String
The provider-assigned unique ID for this managed resource.
jobId String
The unique ID of this job.
pulumiLabels Map<String,String>
The combination of labels configured directly on the resource and default labels configured on the provider.
state String
The current state of the resource, selected from the JobState enum
type String
The type of this job, selected from the JobType enum.
effectiveLabels {[key: string]: string}
id string
The provider-assigned unique ID for this managed resource.
jobId string
The unique ID of this job.
pulumiLabels {[key: string]: string}
The combination of labels configured directly on the resource and default labels configured on the provider.
state string
The current state of the resource, selected from the JobState enum
type string
The type of this job, selected from the JobType enum.
effective_labels Mapping[str, str]
id str
The provider-assigned unique ID for this managed resource.
job_id str
The unique ID of this job.
pulumi_labels Mapping[str, str]
The combination of labels configured directly on the resource and default labels configured on the provider.
state str
The current state of the resource, selected from the JobState enum
type str
The type of this job, selected from the JobType enum.
effectiveLabels Map<String>
id String
The provider-assigned unique ID for this managed resource.
jobId String
The unique ID of this job.
pulumiLabels Map<String>
The combination of labels configured directly on the resource and default labels configured on the provider.
state String
The current state of the resource, selected from the JobState enum
type String
The type of this job, selected from the JobType enum.

Look up Existing FlexTemplateJob Resource

Get an existing FlexTemplateJob resource’s state with the given name, ID, and optional extra properties used to qualify the lookup.

public static get(name: string, id: Input<ID>, state?: FlexTemplateJobState, opts?: CustomResourceOptions): FlexTemplateJob
@staticmethod
def get(resource_name: str,
        id: str,
        opts: Optional[ResourceOptions] = None,
        additional_experiments: Optional[Sequence[str]] = None,
        autoscaling_algorithm: Optional[str] = None,
        container_spec_gcs_path: Optional[str] = None,
        effective_labels: Optional[Mapping[str, str]] = None,
        enable_streaming_engine: Optional[bool] = None,
        ip_configuration: Optional[str] = None,
        job_id: Optional[str] = None,
        kms_key_name: Optional[str] = None,
        labels: Optional[Mapping[str, str]] = None,
        launcher_machine_type: Optional[str] = None,
        machine_type: Optional[str] = None,
        max_workers: Optional[int] = None,
        name: Optional[str] = None,
        network: Optional[str] = None,
        num_workers: Optional[int] = None,
        on_delete: Optional[str] = None,
        parameters: Optional[Mapping[str, str]] = None,
        project: Optional[str] = None,
        pulumi_labels: Optional[Mapping[str, str]] = None,
        region: Optional[str] = None,
        sdk_container_image: Optional[str] = None,
        service_account_email: Optional[str] = None,
        skip_wait_on_job_termination: Optional[bool] = None,
        staging_location: Optional[str] = None,
        state: Optional[str] = None,
        subnetwork: Optional[str] = None,
        temp_location: Optional[str] = None,
        transform_name_mapping: Optional[Mapping[str, str]] = None,
        type: Optional[str] = None) -> FlexTemplateJob
func GetFlexTemplateJob(ctx *Context, name string, id IDInput, state *FlexTemplateJobState, opts ...ResourceOption) (*FlexTemplateJob, error)
public static FlexTemplateJob Get(string name, Input<string> id, FlexTemplateJobState? state, CustomResourceOptions? opts = null)
public static FlexTemplateJob get(String name, Output<String> id, FlexTemplateJobState state, CustomResourceOptions options)
resources:  _:    type: gcp:dataflow:FlexTemplateJob    get:      id: ${id}
name This property is required.
The unique name of the resulting resource.
id This property is required.
The unique provider ID of the resource to lookup.
state
Any extra arguments used during the lookup.
opts
A bag of options that control this resource's behavior.
resource_name This property is required.
The unique name of the resulting resource.
id This property is required.
The unique provider ID of the resource to lookup.
name This property is required.
The unique name of the resulting resource.
id This property is required.
The unique provider ID of the resource to lookup.
state
Any extra arguments used during the lookup.
opts
A bag of options that control this resource's behavior.
name This property is required.
The unique name of the resulting resource.
id This property is required.
The unique provider ID of the resource to lookup.
state
Any extra arguments used during the lookup.
opts
A bag of options that control this resource's behavior.
name This property is required.
The unique name of the resulting resource.
id This property is required.
The unique provider ID of the resource to lookup.
state
Any extra arguments used during the lookup.
opts
A bag of options that control this resource's behavior.
The following state arguments are supported:
AdditionalExperiments List<string>
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
AutoscalingAlgorithm string
The algorithm to use for autoscaling.
ContainerSpecGcsPath string
The GCS path to the Dataflow job Flex Template.

EffectiveLabels Dictionary<string, string>
EnableStreamingEngine Changes to this property will trigger replacement. bool
Immutable. Indicates if the job should use the streaming engine feature.
IpConfiguration string
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
JobId string
The unique ID of this job.
KmsKeyName string
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
Labels Dictionary<string, string>
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
LauncherMachineType string
The machine type to use for launching the job. The default is n1-standard-1.
MachineType string
The machine type to use for the job.
MaxWorkers Changes to this property will trigger replacement. int
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
Name Changes to this property will trigger replacement. string
Immutable. A unique name for the resource, required by Dataflow.
Network string
The network to which VMs will be assigned. If it is not provided, "default" will be used.
NumWorkers Changes to this property will trigger replacement. int
Immutable. The initial number of Google Compute Engine instances for the job.
OnDelete string
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
Parameters Dictionary<string, string>
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
Project Changes to this property will trigger replacement. string
The project in which the resource belongs. If it is not provided, the provider project is used.
PulumiLabels Dictionary<string, string>
The combination of labels configured directly on the resource and default labels configured on the provider.
Region Changes to this property will trigger replacement. string
Immutable. The region in which the created job should run.
SdkContainerImage string
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
ServiceAccountEmail string
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
SkipWaitOnJobTermination bool
StagingLocation string
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
State string
The current state of the resource, selected from the JobState enum
Subnetwork string
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
TempLocation string
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
TransformNameMapping Dictionary<string, string>
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
Type string
The type of this job, selected from the JobType enum.
AdditionalExperiments []string
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
AutoscalingAlgorithm string
The algorithm to use for autoscaling.
ContainerSpecGcsPath string
The GCS path to the Dataflow job Flex Template.

EffectiveLabels map[string]string
EnableStreamingEngine Changes to this property will trigger replacement. bool
Immutable. Indicates if the job should use the streaming engine feature.
IpConfiguration string
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
JobId string
The unique ID of this job.
KmsKeyName string
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
Labels map[string]string
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
LauncherMachineType string
The machine type to use for launching the job. The default is n1-standard-1.
MachineType string
The machine type to use for the job.
MaxWorkers Changes to this property will trigger replacement. int
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
Name Changes to this property will trigger replacement. string
Immutable. A unique name for the resource, required by Dataflow.
Network string
The network to which VMs will be assigned. If it is not provided, "default" will be used.
NumWorkers Changes to this property will trigger replacement. int
Immutable. The initial number of Google Compute Engine instances for the job.
OnDelete string
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
Parameters map[string]string
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
Project Changes to this property will trigger replacement. string
The project in which the resource belongs. If it is not provided, the provider project is used.
PulumiLabels map[string]string
The combination of labels configured directly on the resource and default labels configured on the provider.
Region Changes to this property will trigger replacement. string
Immutable. The region in which the created job should run.
SdkContainerImage string
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
ServiceAccountEmail string
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
SkipWaitOnJobTermination bool
StagingLocation string
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
State string
The current state of the resource, selected from the JobState enum
Subnetwork string
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
TempLocation string
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
TransformNameMapping map[string]string
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
Type string
The type of this job, selected from the JobType enum.
additionalExperiments List<String>
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
autoscalingAlgorithm String
The algorithm to use for autoscaling.
containerSpecGcsPath String
The GCS path to the Dataflow job Flex Template.

effectiveLabels Map<String,String>
enableStreamingEngine Changes to this property will trigger replacement. Boolean
Immutable. Indicates if the job should use the streaming engine feature.
ipConfiguration String
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
jobId String
The unique ID of this job.
kmsKeyName String
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
labels Map<String,String>
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
launcherMachineType String
The machine type to use for launching the job. The default is n1-standard-1.
machineType String
The machine type to use for the job.
maxWorkers Changes to this property will trigger replacement. Integer
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
name Changes to this property will trigger replacement. String
Immutable. A unique name for the resource, required by Dataflow.
network String
The network to which VMs will be assigned. If it is not provided, "default" will be used.
numWorkers Changes to this property will trigger replacement. Integer
Immutable. The initial number of Google Compute Engine instances for the job.
onDelete String
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
parameters Map<String,String>
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
project Changes to this property will trigger replacement. String
The project in which the resource belongs. If it is not provided, the provider project is used.
pulumiLabels Map<String,String>
The combination of labels configured directly on the resource and default labels configured on the provider.
region Changes to this property will trigger replacement. String
Immutable. The region in which the created job should run.
sdkContainerImage String
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
serviceAccountEmail String
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
skipWaitOnJobTermination Boolean
stagingLocation String
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
state String
The current state of the resource, selected from the JobState enum
subnetwork String
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
tempLocation String
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
transformNameMapping Map<String,String>
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
type String
The type of this job, selected from the JobType enum.
additionalExperiments string[]
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
autoscalingAlgorithm string
The algorithm to use for autoscaling.
containerSpecGcsPath string
The GCS path to the Dataflow job Flex Template.

effectiveLabels {[key: string]: string}
enableStreamingEngine Changes to this property will trigger replacement. boolean
Immutable. Indicates if the job should use the streaming engine feature.
ipConfiguration string
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
jobId string
The unique ID of this job.
kmsKeyName string
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
labels {[key: string]: string}
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
launcherMachineType string
The machine type to use for launching the job. The default is n1-standard-1.
machineType string
The machine type to use for the job.
maxWorkers Changes to this property will trigger replacement. number
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
name Changes to this property will trigger replacement. string
Immutable. A unique name for the resource, required by Dataflow.
network string
The network to which VMs will be assigned. If it is not provided, "default" will be used.
numWorkers Changes to this property will trigger replacement. number
Immutable. The initial number of Google Compute Engine instances for the job.
onDelete string
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
parameters {[key: string]: string}
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
project Changes to this property will trigger replacement. string
The project in which the resource belongs. If it is not provided, the provider project is used.
pulumiLabels {[key: string]: string}
The combination of labels configured directly on the resource and default labels configured on the provider.
region Changes to this property will trigger replacement. string
Immutable. The region in which the created job should run.
sdkContainerImage string
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
serviceAccountEmail string
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
skipWaitOnJobTermination boolean
stagingLocation string
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
state string
The current state of the resource, selected from the JobState enum
subnetwork string
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
tempLocation string
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
transformNameMapping {[key: string]: string}
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
type string
The type of this job, selected from the JobType enum.
additional_experiments Sequence[str]
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
autoscaling_algorithm str
The algorithm to use for autoscaling.
container_spec_gcs_path str
The GCS path to the Dataflow job Flex Template.

effective_labels Mapping[str, str]
enable_streaming_engine Changes to this property will trigger replacement. bool
Immutable. Indicates if the job should use the streaming engine feature.
ip_configuration str
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
job_id str
The unique ID of this job.
kms_key_name str
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
labels Mapping[str, str]
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
launcher_machine_type str
The machine type to use for launching the job. The default is n1-standard-1.
machine_type str
The machine type to use for the job.
max_workers Changes to this property will trigger replacement. int
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
name Changes to this property will trigger replacement. str
Immutable. A unique name for the resource, required by Dataflow.
network str
The network to which VMs will be assigned. If it is not provided, "default" will be used.
num_workers Changes to this property will trigger replacement. int
Immutable. The initial number of Google Compute Engine instances for the job.
on_delete str
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
parameters Mapping[str, str]
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
project Changes to this property will trigger replacement. str
The project in which the resource belongs. If it is not provided, the provider project is used.
pulumi_labels Mapping[str, str]
The combination of labels configured directly on the resource and default labels configured on the provider.
region Changes to this property will trigger replacement. str
Immutable. The region in which the created job should run.
sdk_container_image str
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
service_account_email str
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
skip_wait_on_job_termination bool
staging_location str
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
state str
The current state of the resource, selected from the JobState enum
subnetwork str
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
temp_location str
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
transform_name_mapping Mapping[str, str]
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
type str
The type of this job, selected from the JobType enum.
additionalExperiments List<String>
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
autoscalingAlgorithm String
The algorithm to use for autoscaling.
containerSpecGcsPath String
The GCS path to the Dataflow job Flex Template.

effectiveLabels Map<String>
enableStreamingEngine Changes to this property will trigger replacement. Boolean
Immutable. Indicates if the job should use the streaming engine feature.
ipConfiguration String
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
jobId String
The unique ID of this job.
kmsKeyName String
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
labels Map<String>
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
launcherMachineType String
The machine type to use for launching the job. The default is n1-standard-1.
machineType String
The machine type to use for the job.
maxWorkers Changes to this property will trigger replacement. Number
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
name Changes to this property will trigger replacement. String
Immutable. A unique name for the resource, required by Dataflow.
network String
The network to which VMs will be assigned. If it is not provided, "default" will be used.
numWorkers Changes to this property will trigger replacement. Number
Immutable. The initial number of Google Compute Engine instances for the job.
onDelete String
One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.
parameters Map<String>
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
project Changes to this property will trigger replacement. String
The project in which the resource belongs. If it is not provided, the provider project is used.
pulumiLabels Map<String>
The combination of labels configured directly on the resource and default labels configured on the provider.
region Changes to this property will trigger replacement. String
Immutable. The region in which the created job should run.
sdkContainerImage String
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
serviceAccountEmail String
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
skipWaitOnJobTermination Boolean
stagingLocation String
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
state String
The current state of the resource, selected from the JobState enum
subnetwork String
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
tempLocation String
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
transformNameMapping Map<String>
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.
type String
The type of this job, selected from the JobType enum.

Import

This resource does not support import.

To learn more about importing existing cloud resources, see Importing resources.

Package Details

Repository
Google Cloud (GCP) Classic pulumi/pulumi-gcp
License
Apache-2.0
Notes
This Pulumi package is based on the google-beta Terraform Provider.
Google Cloud v8.21.0 published on Wednesday, Mar 5, 2025 by Pulumi
pulumi/pulumi-gcp