Skip to content

Commit 3e59ea9

Browse files
mohamedawnallahgemini-code-assist[bot]
mohamedawnallah
and
gemini-code-assist[bot]
authored
[1/2] sdks/python: enrich data with CloudSQL [PostgreSQL, MySQL, SQLServer] (#34398)
* sdks/python: enrich data with CloudSQL * sdks/python: unit test `CloudSQLEnrichmentHandler` * sdks/python: itest `CloudSQLEnrichmentHandler` * website+sdks: doc `CloudSQLEnrichmentHandler` * sdks/python: address claudevdm feedback (1) * sdks/python: address claudevdm feedback (1)(test) * sdks/python: address claudevdm feedback (1)(itest) * sdks: update doc `CloudSQLEnrichmentHandler` * sdks/python: fix linting issues * website: add missing `enrichment-cloudsql.md` * nits: remove commented code * sdks/python: remove `SQL_TABLE_ID` env variable * sdks/python: fix formatting issues * sdks/python: address claudevdm feedback (2) * sdks/python: address claudevdm feedback (3) * multi: fix pymssql mac os deps * multi: use pytds sql server database adapter In this commit, we use pytds microsoft sqlserver adapter instead of pymssql for one main reason it supports cross-platform compatibility epsecially with macOS so we don't need to install FreeTDS C dependencies. * sdks/python: use `VARCHAR` datatype * sdks/python: address claudevdm feedback (4) * fix setup.py * sdks/python: fix linting issues * sdks/python: connect to beam cloudsql itest instance * .github: trigger postcommits python * sdks/python: update base image requirements * .github: trigger beam postcommit python sdk * examples+transforms: configure sql databases for transform exmaples * apache_beam: fix dependencies * sdks/python: update `get_connector_handler` python docstring * sdks/python: fix linting for cloudsql * .github: trigger post commits python * sdks/python+website: update docs and fix linting issues * CHANGES.md: update release notes * .github: trigger postcommit python CI * sdks/python: fix linting issues * .github: trigger postcommit python CI * sdks/python: fix linting issues * .github: trigger postcommit python CI * sdks/python: fix `examples` CI for cloudSQL enrichment * .github: trigger postcommit python CI * .github/workflows: include `ALLOYDB_PASSWORD` env in precommit python examples * sdks/python: fix linting issues for `cloudsql` * sdks/python: fix linting issues for `enrichment_test.py` Signed-off-by: Mohamed Awnallah <mohamedmohey2352@gmail.com> * .github: trigger postcommit * .github+website+sdks/python: remove non-functional docs * .github: trigger postcommit python CI * sdks/python: generate unique table for itest * .github: enable access to `ALLOYDB_PASSWORD` in precommit py transforms * .github: trigger post commit python * sdks/python: update py39 image requirements * .github: trigger postcommit python * sdks/python: update image requirements * .github: trigger postcommit python * .github: trigger postcommit python * sdks/python: fix local unbound variable * .github: trigger postcommit python * sdks/python: remove params * .github: trigger beam post commit python * sdks/python: experiment with my google cloud * sdks/python: revert `ALLOYDB_PASSWORD` env var in precommit py transform workflow * sdks/python: skip cloudsql tests if `ALLOYDB_PASSWORD` not set * .github: trigger postcommit python * sdks/python: address gemini feedback Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * .github: trigger postcommit python * .github: trigger postcommit python * sdks/python: fix linting issues * sdks/python: fix linting issues * .github: trigger post commit python * sdks/python: fix linting issues * .github: trigger postcommit python * sdks/python: fix linting issues * .github: trigger postcommit python * sdks/python: fix linting issues * .github: trigger postcommit python * sdks/python: fix linting issues * sdks/python: fix SQL injection * CHANGES.md: update release notes * sdks/python: fix formatting issues * sdks/python: fix linting issues * sdks/python: update py image requirements * .github: trigger postcommit python * sdks/python: try to fix python doc linting issues * sdks/python: fix linting issues * .github: trigger postcommit python --------- Signed-off-by: Mohamed Awnallah <mohamedmohey2352@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1 parent 90f8c50 commit 3e59ea9

16 files changed

Lines changed: 2002 additions & 141 deletions

‎.github/trigger_files/beam_PostCommit_Python.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
22
"comment": "Modify this file in a trivial way to cause this test suite to run.",
3-
"modification": 35
3+
"modification": 33
44
}
55

‎.github/workflows/beam_PreCommit_Python_Transforms.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,6 @@ env:
5353
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
5454
GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
5555
GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
56-
ALLOYDB_PASSWORD: ${{ secrets.ALLOYDB_PASSWORD }}
5756

5857
jobs:
5958
beam_PreCommit_Python_Transforms:

‎CHANGES.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,10 @@
7676
* BigtableRead Connector for BeamYaml added with new Config Param ([#35696](https://github.com/apache/beam/pull/35696))
7777
* Introduced a dedicated module for JUnit-based testing support: `sdks/java/testing/junit`, which provides `TestPipelineExtension` for JUnit 5 while maintaining backward compatibility with existing JUnit 4 `TestRule`-based tests (Java) ([#18733](https://github.com/apache/beam/issues/18733), [#35688](https://github.com/apache/beam/pull/35688)).
7878
- To use JUnit 5 with Beam tests, add a test-scoped dependency on `org.apache.beam:beam-sdks-java-testing-junit`.
79+
* Google CloudSQL enrichment handler added (Python) ([#34398](https://github.com/apache/beam/pull/34398)).
80+
Beam now supports data enrichment capabilities using SQL databases, with built-in support for:
81+
- Managed PostgreSQL, MySQL, and Microsoft SQL Server instances on CloudSQL
82+
- Unmanaged SQL database instances not hosted on CloudSQL (e.g., self-hosted or on-premises databases)
7983

8084
## Breaking Changes
8185

0 commit comments

Comments
 (0)