DataprocJob

FieldTypeDescription
apiVersionstringgoogle.kubeform.com/v1alpha1
kindstringDataprocJob
metadataKubernetes meta/v1.ObjectMetaRefer to the Kubernetes API documentation for the fields of the metadata field.
specDataprocJobSpec
statusDataprocJobStatus

DataprocJobSpec

Appears on:DataprocJob, DataprocJobStatus

FieldTypeDescription
providerRefKubernetes core/v1.LocalObjectReference
idstring
driverControlsFilesURIstring(Optional) Output-only. If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
driverOutputResourceURIstring(Optional) Output-only. A URI pointing to the location of the stdout of the job’s driver program
forceDeletebool(Optional)
hadoopConfig[]DataprocJobSpecHadoopConfig(Optional)
hiveConfig[]DataprocJobSpecHiveConfig(Optional)
labelsmap[string]string(Optional) Optional. The labels to associate with this job.
pigConfig[]DataprocJobSpecPigConfig(Optional)
placement[]DataprocJobSpecPlacement
projectstring(Optional)
pysparkConfig[]DataprocJobSpecPysparkConfig(Optional)
reference[]DataprocJobSpecReference(Optional)
regionstring(Optional)
scheduling[]DataprocJobSpecScheduling(Optional) Optional. Job scheduling configuration.
sparkConfig[]DataprocJobSpecSparkConfig(Optional)
sparksqlConfig[]DataprocJobSpecSparksqlConfig(Optional)
status[]DataprocJobSpecStatus(Optional)

DataprocJobSpecHadoopConfig

Appears on:DataprocJobSpec

FieldTypeDescription
archiveUris[]string(Optional)
args[]string(Optional)
fileUris[]string(Optional)
jarFileUris[]string(Optional)
loggingConfig[]DataprocJobSpecHadoopConfigLoggingConfig(Optional) The runtime logging config of the job
mainClassstring(Optional)
mainJarFileURIstring(Optional)
propertiesmap[string]string(Optional)

DataprocJobSpecHadoopConfigLoggingConfig

Appears on:DataprocJobSpecHadoopConfig

FieldTypeDescription
driverLogLevelsmap[string]string(Optional) Optional. The per-package log levels for the driver. This may include ‘root’ package name to configure rootLogger. Examples: ‘com.google = FATAL’, ‘root = INFO’, ‘org.apache = DEBUG’.

DataprocJobSpecHiveConfig

Appears on:DataprocJobSpec

FieldTypeDescription
continueOnFailurebool(Optional)
jarFileUris[]string(Optional)
propertiesmap[string]string(Optional)
queryFileURIstring(Optional)
queryList[]string(Optional)
scriptVariablesmap[string]string(Optional)

DataprocJobSpecPigConfig

Appears on:DataprocJobSpec

FieldTypeDescription
continueOnFailurebool(Optional)
jarFileUris[]string(Optional)
loggingConfig[]DataprocJobSpecPigConfigLoggingConfig(Optional) The runtime logging config of the job
propertiesmap[string]string(Optional)
queryFileURIstring(Optional)
queryList[]string(Optional)
scriptVariablesmap[string]string(Optional)

DataprocJobSpecPigConfigLoggingConfig

Appears on:DataprocJobSpecPigConfig

FieldTypeDescription
driverLogLevelsmap[string]string(Optional) Optional. The per-package log levels for the driver. This may include ‘root’ package name to configure rootLogger. Examples: ‘com.google = FATAL’, ‘root = INFO’, ‘org.apache = DEBUG’.

DataprocJobSpecPlacement

Appears on:DataprocJobSpec

FieldTypeDescription
clusterNamestringThe name of the cluster where the job will be submitted
clusterUUIDstring(Optional) Output-only. A cluster UUID generated by the Cloud Dataproc service when the job is submitted

DataprocJobSpecPysparkConfig

Appears on:DataprocJobSpec

FieldTypeDescription
archiveUris[]string(Optional) Optional. HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip
args[]string(Optional) Optional. The arguments to pass to the driver. Do not include arguments, such as –conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission
fileUris[]string(Optional) Optional. HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks
jarFileUris[]string(Optional) Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks
loggingConfig[]DataprocJobSpecPysparkConfigLoggingConfig(Optional) The runtime logging config of the job
mainPythonFileURIstringRequired. The HCFS URI of the main Python file to use as the driver. Must be a .py file
propertiesmap[string]string(Optional) Optional. A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code
pythonFileUris[]string(Optional) Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip

DataprocJobSpecPysparkConfigLoggingConfig

Appears on:DataprocJobSpecPysparkConfig

FieldTypeDescription
driverLogLevelsmap[string]string(Optional) Optional. The per-package log levels for the driver. This may include ‘root’ package name to configure rootLogger. Examples: ‘com.google = FATAL’, ‘root = INFO’, ‘org.apache = DEBUG’.

DataprocJobSpecReference

Appears on:DataprocJobSpec

FieldTypeDescription
jobIDstring(Optional) The job ID, which must be unique within the project. The job ID is generated by the server upon job submission or provided by the user as a means to perform retries without creating duplicate jobs

DataprocJobSpecScheduling

Appears on:DataprocJobSpec

FieldTypeDescription
maxFailuresPerHourint64(Optional) Maximum number of times per hour a driver may be restarted as a result of driver terminating with non-zero code before job is reported failed.

DataprocJobSpecSparkConfig

Appears on:DataprocJobSpec

FieldTypeDescription
archiveUris[]string(Optional)
args[]string(Optional)
fileUris[]string(Optional)
jarFileUris[]string(Optional)
loggingConfig[]DataprocJobSpecSparkConfigLoggingConfig(Optional) The runtime logging config of the job
mainClassstring(Optional)
mainJarFileURIstring(Optional)
propertiesmap[string]string(Optional)

DataprocJobSpecSparkConfigLoggingConfig

Appears on:DataprocJobSpecSparkConfig

FieldTypeDescription
driverLogLevelsmap[string]string(Optional) Optional. The per-package log levels for the driver. This may include ‘root’ package name to configure rootLogger. Examples: ‘com.google = FATAL’, ‘root = INFO’, ‘org.apache = DEBUG’.

DataprocJobSpecSparksqlConfig

Appears on:DataprocJobSpec

FieldTypeDescription
jarFileUris[]string(Optional)
loggingConfig[]DataprocJobSpecSparksqlConfigLoggingConfig(Optional) The runtime logging config of the job
propertiesmap[string]string(Optional)
queryFileURIstring(Optional)
queryList[]string(Optional)
scriptVariablesmap[string]string(Optional)

DataprocJobSpecSparksqlConfigLoggingConfig

Appears on:DataprocJobSpecSparksqlConfig

FieldTypeDescription
driverLogLevelsmap[string]string(Optional) Optional. The per-package log levels for the driver. This may include ‘root’ package name to configure rootLogger. Examples: ‘com.google = FATAL’, ‘root = INFO’, ‘org.apache = DEBUG’.

DataprocJobSpecStatus

Appears on:DataprocJobSpec

FieldTypeDescription
detailsstring(Optional) Output-only. Optional job state details, such as an error description if the state is ERROR
statestring(Optional) Output-only. A state message specifying the overall job state
stateStartTimestring(Optional) Output-only. The time when this state was entered
substatestring(Optional) Output-only. Additional state information, which includes status reported by the agent

DataprocJobStatus

Appears on:DataprocJob

FieldTypeDescription
observedGenerationint64(Optional) Resource generation, which is updated on mutation by the API Server.
outputDataprocJobSpec(Optional)
statekubeform.dev/kubeform/apis/base/v1alpha1.State(Optional)
phasePhase(Optional)

Phase(string alias)

Appears on:DataprocJobStatus