Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/docs-site/docs/templates/kafka-to-helixdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,14 @@ Now the template is built and hosted your GCS path. you can pass GCS template fi

If you have an Apache Flink standalone cluster, you can submit the template as a job using the prebuilt Docker image.

> **Note:** Don’t have a remote Flink cluster? No problem!
> You can quickly spin up Flink on your laptop in just [3 simple steps](https://ganeshsivakumar.github.io/langchain-beam/docs/tutorials/setup-flink/) and run templates locally.

#### Submitting Job

Once your Flink cluster is up and running, you can submit a template as a job using Docker.
Run the following command, adjusting the pipeline options as needed:

```
docker run --rm \
-e FLINK_MASTER=host.docker.internal:8081 \
Expand All @@ -146,7 +154,7 @@ Refer to the Flink version [compatibility matrix](https://beam.apache.org/docume

- The container downloads the appropriate .jar file from GCS based on ***FLINK_VERSION*** (your flink cluster version) with correct beam and runner dependencies

- It uses the Flink CLI (flink run) to submit the job to the Flink cluster (as specified by `FLINK_MASTER`)
- It uses the Flink CLI (flink run) to submit the job to the Flink cluster (as specified by ***FLINK_MASTER*** - Flink cluster Url )

- All dependencies—including Java 17 and the Flink CLI—are preinstalled in the image, so you don’t need to set up anything else.

Expand Down
11 changes: 10 additions & 1 deletion docs/docs-site/docs/templates/kafka-to-pinecone.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,15 @@ Now the template is built and hosted your GCS path. you can pass GCS template fi

If you have an Apache Flink standalone cluster, you can submit the template as a job using the prebuilt Docker image.

> **Note:** Don’t have a remote Flink cluster? No problem!
> You can quickly spin up Flink on your laptop in just [3 simple steps](https://ganeshsivakumar.github.io/langchain-beam/docs/tutorials/setup-flink/) and run templates locally.

#### Submitting Job

Once your Flink cluster is up and running, you can submit a template as a job using Docker.
Run the following command, adjusting the pipeline options as needed:


```
docker run --rm \
-e FLINK_MASTER=host.docker.internal:8081 \
Expand Down Expand Up @@ -137,7 +146,7 @@ Refer to the Flink version [compatibility matrix](https://beam.apache.org/docume

- The container downloads the appropriate .jar file from GCS based on ***FLINK_VERSION*** (your flink cluster version) with correct beam and runner dependencies

- It uses the Flink CLI (flink run) to submit the job to the Flink cluster (as specified by `FLINK_MASTER`)
- It uses the Flink CLI (flink run) to submit the job to the Flink cluster (as specified by ***FLINK_MASTER*** - Flink cluster Url )

- All dependencies—including Java 17 and the Flink CLI—are preinstalled in the image, so you don’t need to set up anything else.

Expand Down
49 changes: 49 additions & 0 deletions docs/docs-site/docs/tutorials/setup-flink.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# How to run Apache Flink Locally

[Apache Flink](https://flink.apache.org/) is a powerful stream-processing engine. It can also run [Apache Beam](https://beam.apache.org/) pipelines and templates using the Flink Runner. This short guide walks you through setting up Flink on your laptop in just a few steps.


#### Step 1: Download Flink 1.19

Use `wget` to download the latest Flink 1.19 release (Scala 2.12 build):

```sh
wget https://dlcdn.apache.org/flink/flink-1.19.3/flink-1.19.3-bin-scala_2.12.tgz
```

#### Step 2: Extract the Archive

Unpack the downloaded archive:

```sh
tar -xzf flink-1.19.3-bin-scala_2.12.tgz
```

#### Step 3: Start the Flink Cluster

Navigate into the extracted directory and start a local cluster:

```sh
cd flink-1.19.3

# Start the cluster
./bin/start-cluster.sh
```

#### Step 4: Verify the Cluster

Once started, open the **Flink Web UI** in your browser:

👉 [http://localhost:8081/](http://localhost:8081/)

If the UI loads, your local Flink cluster is up and running! 🎉
Next, try running some [LangBeam templates](https://ganeshsivakumar.github.io/langchain-beam/docs/category/templates/) on your Flink cluster 😃


#### Step 5: Stop the Cluster

When you’re done, stop the cluster to free up resources:

```sh
./bin/stop-cluster.sh
```