[Bug]: beam.io.gcp.bigquery.ReadFromBigQuery failing with permission issue

[ssu1276]

What happened?

Hello,
I am running a GCP dataflow job programmed in python using the latest version of apache beam 2.67. I am using lib beam.io.gcp.bigquery.ReadFromBigQuery to read data from Big Query. My job is getting failed with a permission issue when i am using a different project(other than where data flow job is running) in query to fetch the data and the same job is running fine with version 2.65. I was getting failed with 2.66 as well.
Thanks in advance ...!

Issue Priority

Priority: 0 (outage / urgent vulnerability)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[shahar1]

I opened a related issue (feature req.) last year: #31126 (unfortunately I didn't mange to work on it since then, I might soon).

Although you say that it did work in 2.65 - which sounds odd. AFAIK, the BQ job is created on the same project where the BQ dataset is - so to make it work, the jobUser (+readSessionUser) roles should be given in that project for the service account from the other one.

What permission issue did you get?

[ssu1276]

Yes it was running fine before Apache Beam version 2.66. And i am running a query in this task which contains join of two table one from the project where dataflow job is running and one from other.
I am getting below error which shows doesn't have permission to query table:

"errors": [
      {
        "message": "Access Denied: Table <_Other Project Table_>: User does not have permission to query table <_Other Project Table_>, or perhaps it does not exist.",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}
> [while running 'read.deleted.profiles.from-gbq/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitWithSizing-ptransform-38']

[shahar1]

Yes it was running fine before Apache Beam version 2.66. And i am running a query in this task which contains join of two table one from the project where dataflow job is running and one from other. I am getting below error which shows doesn't have permission to query table:

"errors": [
      {
        "message": "Access Denied: Table <_Other Project Table_>: User does not have permission to query table <_Other Project Table_>, or perhaps it does not exist.",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}
> [while running 'read.deleted.profiles.from-gbq/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitWithSizing-ptransform-38']

Thanks, could you please share a full stacktrace within the apache beam source code? It will make it easier to correlate with specific changes.

[ssu1276]

Below is the complete error log:

Workflow failed. Causes: S34:read.deleted.profiles.from-gbq/Read/Impulse+read.deleted.profiles.from-gbq/Read/EmitSource+read.deleted.profiles.from-gbq/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction+read.deleted.profiles.from-gbq/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitWithSizing failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#job-failed-four-times. The work item was attempted on these workers:

      Root cause: Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1498, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 684, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1673, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1732, in process
    for part, size in self.restriction_provider.split_and_size(
  File "/usr/local/lib/python3.11/site-packages/apache_beam/transforms/core.py", line 334, in split_and_size
    for part in self.split(element, restriction):
  File "/usr/local/lib/python3.11/site-packages/apache_beam/io/iobase.py", line 1868, in split
    estimated_size = restriction.source().estimate_size()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery.py", line 746, in estimate_size
    job = bq._start_query_job(
          ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/utils/retry.py", line 300, in wrapper
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 652, in _start_query_job
    return self._start_job(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 592, in _start_job
    response = self.client.jobs.Insert(request, upload=upload)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 347, in Insert
    return self._RunMethod(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
    return self.ProcessHttpResponse(method_config, http_response, request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
    self.__ProcessHttpResponse(method_config, http_response, request))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apitools/base/py/base_api.py", line 603, in __ProcessHttpResponse
    raise exceptions.HttpError.FromResponse(
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/researcher-360-qa-aa6acd60/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Tue, 26 Aug 2025 06:18:17 GMT', 'server': 'ESF', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '403', 'content-length': '708', '-content-encoding': 'gzip'}>, content <{
  "error": {
    "code": 403,
    "message": "Access Denied: Table <Other Project Table>: User does not have permission to query table <Other Project Table>, or perhaps it does not exist.",
    "errors": [
      {
        "message": "Access Denied: Table <Other Project Table>: User does not have permission to query table <Other Project Table>, or perhaps it does not exist.",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py", line 313, in _execute
    response = task()
               ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py", line 387, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py", line 659, in do_instruction
    return getattr(self, request_type)(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py", line 697, in process_bundle
    bundle_processor.process_bundle(instruction_id))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1274, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 237, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 568, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 570, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 260, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 263, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 951, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 952, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1500, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1588, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1498, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 684, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1683, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1796, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 263, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 951, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 952, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1500, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1588, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1498, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 684, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1683, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1796, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 263, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 951, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 952, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1500, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1609, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1498, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 684, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1673, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1732, in process
    for part, size in self.restriction_provider.split_and_size(
  File "/usr/local/lib/python3.11/site-packages/apache_beam/transforms/core.py", line 334, in split_and_size
    for part in self.split(element, restriction):
  File "/usr/local/lib/python3.11/site-packages/apache_beam/io/iobase.py", line 1868, in split
    estimated_size = restriction.source().estimate_size()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery.py", line 746, in estimate_size
    job = bq._start_query_job(
          ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/utils/retry.py", line 300, in wrapper
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 652, in _start_query_job
    return self._start_job(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 592, in _start_job
    response = self.client.jobs.Insert(request, upload=upload)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 347, in Insert
    return self._RunMethod(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
    return self.ProcessHttpResponse(method_config, http_response, request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
    self.__ProcessHttpResponse(method_config, http_response, request))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/apitools/base/py/base_api.py", line 603, in __ProcessHttpResponse
    raise exceptions.HttpError.FromResponse(
RuntimeError: apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/researcher-360-qa-aa6acd60/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Tue, 26 Aug 2025 06:18:17 GMT', 'server': 'ESF', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '403', 'content-length': '708', '-content-encoding': 'gzip'}>, content <{
  "error": {
    "code": 403,
    "message": "Access Denied: Table <Other Project Table>: User does not have permission to query table <Other Project Table>, or perhaps it does not exist.",
    "errors": [
      {
        "message": "Access Denied: Table <Other Project Table>: User does not have permission to query table <Other Project Table>, or perhaps it does not exist.",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}
> [while running 'read.deleted.profiles.from-gbq/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitWithSizing-ptransform-38']

      Worker ID: sn-honours-t1-customers-p-08252313-7kv3-harness-j3d9

Do you need any other information?

[liferoad]

Does your Dataflow service account have the sufficient permissions?

[ssu1276]

Yes, it has sufficient permissions. Dataflow job is running fine now after degrading apache beam to 2.65 now.

[liferoad]

I do not see any clear code changes between 2.65.0 and 2.66.0 that could cause this permission issue. Can you open a GCP support ticket?

[ssu1276]

Sure let me raise...