https://flink.apache.org/ logo
Join Slack
Powered by
I tried building a docker image with this Dockerfi...
# troubleshooting
a
I tried building a docker image with this Dockerfile.
Copy code
FROM flink:1.17.1


# install python3 and pip3
RUN apt-get update -y && \
apt-get install -y python3 python3-pip python3-dev && rm -rf /var/lib/apt/lists/*
RUN ln -s /usr/bin/python3 /usr/bin/python

# install PyFlink
RUN pip3 install apache-flink==1.17.1
However, I received this error. What do I need to change? It's working fine on my Linux machine but receiving this error on my Mac 
#0 35.81  Downloading pemja-0.3.0.tar.gz (48 kB)
#0 35.81 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.5/48.5 KB 14.6 MB/s eta 00000 #0 35.90 Installing build dependencies: started #0 39.29 Installing build dependencies: finished with status 'done' #0 39.30 Getting requirements to build wheel: started #0 39.36 Getting requirements to build wheel: finished with status 'error' #0 39.37 error: subprocess-exited-with-error #0 39.37 #0 39.37 × Getting requirements to build wheel did not run successfully. #0 39.37 │ exit code: 255 #0 39.37 ╰─> [1 lines of output] #0 39.37 Include folder should be at '/opt/java/openjdk/include' but doesn't exist. Please check you've installed the JDK properly. #0 39.37 [end of output] #0 39.37 #0 39.37 note: This error originates from a subprocess, and is likely not a problem with pip. #0 39.37 error: subprocess-exited-with-error #0 39.37 #0 39.37 × Getting requirements to build wheel did not run successfully. #0 39.37 │ exit code: 255 #0 39.37 ╰─> See above for output. #0 39.37 #0 39.37 note: This error originates from a subprocess, and is likely not a problem with pip.
The issue occurred as Docker was selecting a different OS config that is incompatible with PyFlink.
d
@Arijit Dasgupta were you able to resolve this on Mac? I'm facing the same issue. I also found https://www.mail-archive.com/user@flink.apache.org/msg48255.html, which suggests you need to install JDK explicitly, but I'm yet to try to figure that out.
I think I was able to resolve it by creating a copy of the Flink image based on 
eclipse-temurin:8-jdk-jammy
(instead of 
eclipse-temurin:8-jre-jammy
). It works, although I'm not sure this should be the recommended way:
Copy code
# Build the "official" Flink 1.17.1 image using
# <https://github.com/apache/flink-docker/blob/abc36dd88483a221c0f5495c742bd95c349e9ac2/1.17/scala_2.12-java8-ubuntu/Dockerfile>
# but switch the parent image variant to include JDK
FROM eclipse-temurin:8-jdk-jammy AS builder

# Install dependencies
RUN set -ex; \
  apt-get update; \
  apt-get -y install gpg libsnappy1v5 gettext-base libjemalloc-dev; \
  rm -rf /var/lib/apt/lists/*

# Grab gosu for easy step-down from root
ENV GOSU_VERSION 1.11
RUN set -ex; \
  wget -nv -O /usr/local/bin/gosu "<https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg> --print-architecture)"; \
  wget -nv -O /usr/local/bin/gosu.asc "<https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg> --print-architecture).asc"; \
  export GNUPGHOME="$(mktemp -d)"; \
  for server in <http://ha.pool.sks-keyservers.net|ha.pool.sks-keyservers.net> $(shuf -e \
                          <hkp://p80.pool.sks-keyservers.net:80> \
                          <http://keyserver.ubuntu.com|keyserver.ubuntu.com> \
                          <hkp://keyserver.ubuntu.com:80> \
                          <http://pgp.mit.edu|pgp.mit.edu>) ; do \
      gpg --batch --keyserver "$server" --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 && break || : ; \
  done && \
  gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \
  gpgconf --kill all; \
  rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc; \
  chmod +x /usr/local/bin/gosu; \
  gosu nobody true

# Configure Flink version
ENV FLINK_TGZ_URL=<https://www.apache.org/dyn/closer.cgi?action=download&filename=flink/flink-1.17.1/flink-1.17.1-bin-scala_2.12.tgz> \
    FLINK_ASC_URL=<https://www.apache.org/dist/flink/flink-1.17.1/flink-1.17.1-bin-scala_2.12.tgz.asc> \
    GPG_KEY=8D56AE6E7082699A4870750EA4E8C4C05EE6861F \
    CHECK_GPG=true

# Prepare environment
ENV FLINK_HOME=/opt/flink
ENV PATH=$FLINK_HOME/bin:$PATH
RUN groupadd --system --gid=9999 flink && \
    useradd --system --home-dir $FLINK_HOME --uid=9999 --gid=flink flink
WORKDIR $FLINK_HOME

# Install Flink
RUN set -ex; \
  wget -nv -O flink.tgz "$FLINK_TGZ_URL"; \
  \
  if [ "$CHECK_GPG" = "true" ]; then \
    wget -nv -O flink.tgz.asc "$FLINK_ASC_URL"; \
    export GNUPGHOME="$(mktemp -d)"; \
    for server in <http://ha.pool.sks-keyservers.net|ha.pool.sks-keyservers.net> $(shuf -e \
                            <hkp://p80.pool.sks-keyservers.net:80> \
                            <http://keyserver.ubuntu.com|keyserver.ubuntu.com> \
                            <hkp://keyserver.ubuntu.com:80> \
                            <http://pgp.mit.edu|pgp.mit.edu>) ; do \
        gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ; \
    done && \
    gpg --batch --verify flink.tgz.asc flink.tgz; \
    gpgconf --kill all; \
    rm -rf "$GNUPGHOME" flink.tgz.asc; \
  fi; \
  \
  tar -xf flink.tgz --strip-components=1; \
  rm flink.tgz; \
  \
  chown -R flink:flink .; \
  \
  # Replace default REST/RPC endpoint bind address to use the container's network interface \
  sed -i 's/rest.address: localhost/rest.address: 0.0.0.0/g' $FLINK_HOME/conf/flink-conf.yaml; \
  sed -i 's/rest.bind-address: localhost/rest.bind-address: 0.0.0.0/g' $FLINK_HOME/conf/flink-conf.yaml; \
  sed -i 's/jobmanager.bind-host: localhost/jobmanager.bind-host: 0.0.0.0/g' $FLINK_HOME/conf/flink-conf.yaml; \
  sed -i 's/taskmanager.bind-host: localhost/taskmanager.bind-host: 0.0.0.0/g' $FLINK_HOME/conf/flink-conf.yaml; \
  sed -i '/taskmanager.host: localhost/d' $FLINK_HOME/conf/flink-conf.yaml;

# Configure container
COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 6123 8081
CMD ["help"]

# Build a custom image which has Python and PyFlink prepared, following
# <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/standalone/docker/#using-flink-python-on-docker>
FROM builder

# install python3 and pip3
RUN apt-get update -y && \
apt-get install -y python3 python3-pip python3-dev && rm -rf /var/lib/apt/lists/*
RUN ln -s /usr/bin/python3 /usr/bin/python

# install PyFlink
RUN pip3 install apache-flink==1.17.1
a
@Deepyaman Datta Thank you for the idea and docker file script. Could advise to use the below entry point as well:
Copy code
#!/bin/bash

set -e

# Initialize Flink JobManager
if [ "$1" = 'jobmanager' ]; then
    echo "Starting Flink JobManager"
    exec /opt/flink/bin/jobmanager.sh start-foreground "$@"
fi

# Initialize Flink TaskManager
if [ "$1" = 'taskmanager' ]; then
    echo "Starting Flink TaskManager"
    exec /opt/flink/bin/taskmanager.sh start-foreground "$@"
fi

# Fallback for custom commands
exec "$@"
🙌 1
4 Views
Open in Slack
PreviousNext
Powered by