How can you use BigQuery custom tables and datasets in your future analysis projects

incremental_lineage boolean When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run. True env string The environment that all assets produced by this connector belong to PROD platform string The platform that this source connects to None platform_instance string The instance of the platform that all assets produced by this recipe belong to None options Dict {} include_views boolean Whether views should be ingested. True include_tables boolean Whether tables should be ingested. True bucket_duration enum(BucketDuration) Size of the time window to aggregate usage stats.. Allowed symbols are DAY, HOUR DAY end_time string Latest date of usage to consider. Default: Current time in UTC None start_time string Earliest date of usage to consider. Default: Last full day in UTC (or hour, depending on bucket_duration) None rate_limit boolean Should we rate limit requests made to API. False requests_per_min integer Used to control number of API calls made per min. Only used when rate_limit is set to True. 60 temp_table_dataset_prefix string If you are creating temp tables in a dataset with a particular prefix you can use this config to set the prefix for the dataset. This is to support workflows from before bigquery's introduction of temp tables. By default we use _ because of datasets that begin with an underscore are hidden by default https://cloud.google.com/bigquery/docs/datasets#dataset-naming. _ sharded_table_pattern string The regex pattern to match sharded tables and group as one table. This is a very low level config parameter, only change if you know what you are doing, ((.+)[_$])?(\d{8})$ scheme string bigquery project_id string [deprecated] Use project_id_pattern instead. None log_page_size integer The number of log item will be queried per page for lineage collection 1000 extra_client_options Dict Additional options to pass to google.cloud.logging_v2.client.Client. {} include_table_lineage boolean Option to enable/disable lineage generation. Is enabled by default. True max_query_duration number Correction to pad start_time and end_time with. For handling the case where the read happens within our time range but the query completion event is delayed and happens after the configured end time. 900.0 bigquery_audit_metadata_datasets Array of string A list of datasets that contain a table named cloudaudit_googleapis_com_data_access which contain BigQuery audit logs, specifically, those containing BigQueryAuditMetadata. It is recommended that the project of the dataset is also specified, for example, projectA.datasetB. None use_exported_bigquery_audit_metadata boolean When configured, use BigQueryAuditMetadata in bigquery_audit_metadata_datasets to compute lineage information. False use_date_sharded_audit_log_tables boolean Whether to read date sharded tables or time partitioned tables when extracting usage from exported audit logs. False use_v2_audit_metadata boolean Whether to ingest logs using the v2 format. False upstream_lineage_in_report boolean Useful for debugging lineage information. Set to True to see the raw lineage created internally. False include_usage_statistics boolean Generate usage statistic True capture_table_label_as_tag boolean Capture BigQuery table labels as tag False debug_include_full_payloads boolean Include full payload into events. It is only for debugging and internal use. False number_of_datasets_process_in_batch integer Number of table queried in batch when getting metadata. This is a low level config property which should be touched with care. This restriction is needed because we query partitions system view which throws error if we try to touch too many tables. 80 column_limit integer Maximum number of columns to process in a table. This is a low level config property which should be touched with care. This restriction is needed because excessively wide tables can result in failure to ingest the schema. 300 lineage_use_sql_parser boolean Experimental. Use sql parser to resolve view/table lineage. If there is a view being referenced then bigquery sends both the view as well as underlying tablein the references. There is no distinction between direct/base objects accessed. So doing sql parsing to ensure we only use direct objects accessed for lineage. False lineage_parse_view_ddl boolean Sql parse view ddl to get lineage. True convert_urns_to_lowercase boolean Convert urns to lowercase. False stateful_ingestion SQLAlchemyStatefulIngestionConfig (see below for fields) stateful_ingestion.enabled boolean The type of the ingestion state provider registered with datahub. False stateful_ingestion.max_checkpoint_state_size integer The maximum size of the checkpoint state in bytes. Default is 16MB 16777216 stateful_ingestion.state_provider DynamicTypedStateProviderConfig (see below for fields) The ingestion state provider configuration. stateful_ingestion.state_provider.type ❓ (required if stateful_ingestion.state_provider is set) string The type of the state provider to use. For DataHub use datahub None stateful_ingestion.state_provider.config Generic dict The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19). None stateful_ingestion.ignore_old_state boolean If set to True, ignores the previous checkpoint state. False stateful_ingestion.ignore_new_state boolean If set to True, ignores the current checkpoint state. False stateful_ingestion.remove_stale_metadata boolean Soft-deletes the entities of type in the last successful run but missing in the current run with stateful_ingestion enabled. True stateful_ingestion.fail_safe_threshold number Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'. 20.0 schema_pattern AllowDenyPattern (see below for fields) Regex patterns for schemas to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics' {'allow': ['.*'], 'deny': [], 'ignoreCase': True} schema_pattern.allow Array of string List of regex patterns to include in ingestion ['.*'] schema_pattern.deny Array of string List of regex patterns to exclude from ingestion. [] schema_pattern.ignoreCase boolean Whether to ignore case sensitivity during pattern matching. True table_pattern AllowDenyPattern (see below for fields) Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*' {'allow': ['.*'], 'deny': [], 'ignoreCase': True} table_pattern.allow Array of string List of regex patterns to include in ingestion ['.*'] table_pattern.deny Array of string List of regex patterns to exclude from ingestion. [] table_pattern.ignoreCase boolean Whether to ignore case sensitivity during pattern matching. True view_pattern AllowDenyPattern (see below for fields) Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*' {'allow': ['.*'], 'deny': [], 'ignoreCase': True} view_pattern.allow Array of string List of regex patterns to include in ingestion ['.*'] view_pattern.deny Array of string List of regex patterns to exclude from ingestion. [] view_pattern.ignoreCase boolean Whether to ignore case sensitivity during pattern matching. True profile_pattern AllowDenyPattern (see below for fields) Regex patterns to filter tables for profiling during ingestion. Allowed by the table_pattern. {'allow': ['.*'], 'deny': [], 'ignoreCase': True} profile_pattern.allow Array of string List of regex patterns to include in ingestion ['.*'] profile_pattern.deny Array of string List of regex patterns to exclude from ingestion. [] profile_pattern.ignoreCase boolean Whether to ignore case sensitivity during pattern matching. True domain Dict[str, AllowDenyPattern] Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified. {} domain.key.allow Array of string List of regex patterns to include in ingestion ['.*'] domain.key.deny Array of string List of regex patterns to exclude from ingestion. [] domain.key.ignoreCase boolean Whether to ignore case sensitivity during pattern matching. True profiling GEProfilingConfig (see below for fields) {'enabled': False, 'limit': None, 'offset': None, 'report_dropped_profiles': False, 'turn_off_expensive_profiling_metrics': False, 'profile_table_level_only': False, 'include_field_null_count': True, 'include_field_min_value': True, 'include_field_max_value': True, 'include_field_mean_value': True, 'include_field_median_value': True, 'include_field_stddev_value': True, 'include_field_quantiles': False, 'include_field_distinct_value_frequencies': False, 'include_field_histogram': False, 'include_field_sample_values': True, 'max_number_of_fields_to_profile': None, 'profile_if_updated_since_days': 1, 'profile_table_size_limit': 1, 'profile_table_row_limit': 50000, 'max_workers': 10, 'query_combiner_enabled': True, 'catch_exceptions': True, 'partition_profiling_enabled': True, 'bigquery_temp_table_schema': None, 'partition_datetime': None} profiling.enabled boolean Whether profiling should be done. False profiling.limit integer Max number of documents to profile. By default, profiles all documents. None profiling.offset integer Offset in documents to profile. By default, uses no offset. None profiling.report_dropped_profiles boolean Whether to report datasets or dataset columns which were not profiled. Set to True for debugging purposes. False profiling.turn_off_expensive_profiling_metrics boolean Whether to turn off expensive profiling or not. This turns off profiling for quantiles, distinct_value_frequencies, histogram & sample_values. This also limits maximum number of fields being profiled to 10. False profiling.profile_table_level_only boolean Whether to perform profiling at table-level only, or include column-level profiling as well. False profiling.include_field_null_count boolean Whether to profile for the number of nulls for each column. True profiling.include_field_min_value boolean Whether to profile for the min value of numeric columns. True profiling.include_field_max_value boolean Whether to profile for the max value of numeric columns. True profiling.include_field_mean_value boolean Whether to profile for the mean value of numeric columns. True profiling.include_field_median_value boolean Whether to profile for the median value of numeric columns. True profiling.include_field_stddev_value boolean Whether to profile for the standard deviation of numeric columns. True profiling.include_field_quantiles boolean Whether to profile for the quantiles of numeric columns. False profiling.include_field_distinct_value_frequencies boolean Whether to profile for distinct value frequencies. False profiling.include_field_histogram boolean Whether to profile for the histogram for numeric fields. False profiling.include_field_sample_values boolean Whether to profile for the sample values for all columns. True profiling.max_number_of_fields_to_profile integer A positive integer that specifies the maximum number of columns to profile for any table. None implies all columns. The cost of profiling goes up significantly as the number of columns to profile goes up. None profiling.profile_if_updated_since_days number Profile table only if it has been updated since these many number of days. If set to null, no constraint of last modified time for tables to profile. Supported only in snowflake, snowflake-beta and BigQuery. 1 profiling.profile_table_size_limit integer Profile tables only if their size is less then specified GBs. If set to null, no limit on the size of tables to profile. Supported only in snowflake-beta and BigQuery 1 profiling.profile_table_row_limit integer Profile tables only if their row count is less then specified count. If set to null, no limit on the row count of tables to profile. Supported only in snowflake-beta and BigQuery 50000 profiling.max_workers integer Number of worker threads to use for profiling. Set to 1 to disable. 10 profiling.query_combiner_enabled boolean This feature is still experimental and can be disabled if it causes issues. Reduces the total number of queries issued and speeds up profiling by dynamically combining SQL queries where possible. True profiling.catch_exceptions boolean True profiling.partition_profiling_enabled boolean True profiling.bigquery_temp_table_schema string On bigquery for profiling partitioned tables needs to create temporary views. You have to define a dataset where these will be created. Views will be cleaned up after profiler runs. (Great expectation tech details about this (https://legacy.docs.greatexpectations.io/en/0.9.0/reference/integrations/bigquery.html#custom-queries-with-sql-datasource). None profiling.partition_datetime string For partitioned datasets profile only the partition which matches the datetime or profile the latest one if not set. Only Bigquery supports this. None credential BigQueryCredential (see below for fields) BigQuery credential informations credential.project_id ❓ (required if credential is set) string Project id to set the credentials None credential.private_key_id ❓ (required if credential is set) string Private key id None credential.private_key ❓ (required if credential is set) string Private key in a form of '-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n' None credential.client_email ❓ (required if credential is set) string Client email None credential.client_id ❓ (required if credential is set) string Client Id None credential.auth_uri string Authentication uri https://accounts.google.com/o/oauth2/auth credential.token_uri string Token uri https://oauth2.googleapis.com/token credential.auth_provider_x509_cert_url string Auth provider x509 certificate url https://www.googleapis.com/oauth2/v1/certs credential.type string Authentication type service_account credential.client_x509_cert_url string If not set it will be default to https://www.googleapis.com/robot/v1/metadata/x509/client_email None project_id_pattern AllowDenyPattern (see below for fields) Regex patterns for project_id to filter in ingestion. {'allow': ['.*'], 'deny': [], 'ignoreCase': True} project_id_pattern.allow Array of string List of regex patterns to include in ingestion ['.*'] project_id_pattern.deny Array of string List of regex patterns to exclude from ingestion. [] project_id_pattern.ignoreCase boolean Whether to ignore case sensitivity during pattern matching. True usage BigQueryUsageConfig (see below for fields) Usage related configs {'bucket_duration': <BucketDuration.DAY: 'DAY'>, 'end_time': datetime.datetime(2022, 11, 4, 22, 39, 13, 155142, tzinfo=datetime.timezone.utc), 'start_time': datetime.datetime(2022, 11, 3, 0, 0, tzinfo=datetime.timezone.utc), 'top_n_queries': 10, 'user_email_pattern': {'allow': ['.*'], 'deny': [], 'ignoreCase': True}, 'include_operational_stats': True, 'include_read_operational_stats': False, 'format_sql_queries': False, 'include_top_n_queries': True, 'query_log_delay': None, 'max_query_duration': datetime.timedelta(seconds=900)} usage.bucket_duration enum(BucketDuration) Size of the time window to aggregate usage stats.. Allowed symbols are DAY, HOUR DAY usage.end_time string Latest date of usage to consider. Default: Current time in UTC None usage.start_time string Earliest date of usage to consider. Default: Last full day in UTC (or hour, depending on bucket_duration) None usage.top_n_queries integer Number of top queries to save to each table. 10 usage.user_email_pattern AllowDenyPattern (see below for fields) regex patterns for user emails to filter in usage. {'allow': ['.*'], 'deny': [], 'ignoreCase': True} usage.user_email_pattern.allow Array of string List of regex patterns to include in ingestion ['.*'] usage.user_email_pattern.deny Array of string List of regex patterns to exclude from ingestion. [] usage.user_email_pattern.ignoreCase boolean Whether to ignore case sensitivity during pattern matching. True usage.include_operational_stats boolean Whether to display operational stats. True usage.include_read_operational_stats boolean Whether to report read operational stats. Experimental. False usage.format_sql_queries boolean Whether to format sql queries False usage.include_top_n_queries boolean Whether to ingest the top_n_queries. True usage.query_log_delay integer To account for the possibility that the query event arrives after the read event in the audit logs, we wait for at least query_log_delay additional events to be processed before attempting to resolve BigQuery job information from the logs. If query_log_delay is None, it gets treated as an unlimited delay, which prioritizes correctness at the expense of memory usage. None usage.max_query_duration number Correction to pad start_time and end_time with. For handling the case where the read happens within our time range but the query completion event is delayed and happens after the configured end time. 900.0 dataset_pattern AllowDenyPattern (see below for fields) Regex patterns for dataset to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics' {'allow': ['.*'], 'deny': [], 'ignoreCase': True} dataset_pattern.allow Array of string List of regex patterns to include in ingestion ['.*'] dataset_pattern.deny Array of string List of regex patterns to exclude from ingestion. [] dataset_pattern.ignoreCase boolean Whether to ignore case sensitivity during pattern matching. True

How can you use BigQuery custom tables and datasets?

How do you use BigQuery in Google Analytics?

How do you use BigQuery effectively?

What can I do with Google BigQuery?