Anyone know if PyFlink UDAF in AWS Kinesis Analytics support Apache Flink #troubleshooting

Anyone know if PyFlink UDAF in AWS Kinesis Analyti...

Jeff Levesque

07/29/2023, 12:06 AM

Anyone know if PyFlink UDAF in AWS Kinesis Analytics supports boto3 invocations? I tried to take the AWS sample UDF and add some python logic that says if x > y, then

client.publish(TargetArn='xxx',Subject='xxx',Message='xxx')

. This seems to work when I run PyFlink locally in PyCharm. When I bundle the application as a zip file, then upload to AWS Kinesis Analytics, the application runs non-stop, no exceptions, and nothing in the logs (configured to log with

INFO

debug level) regarding boto3 and sns. I created an AWS support case almost 4 months ago. I asked if "AWS Service (Kinesis Analytics) supports other AWS services (boto3)", and I got a reference to AWS star principle, and then another support engineer said that code support is not supported, then another support engineer took almost 2 months to say that UDAF doesn't support

s3:PutObject

for reasons because he saw another case where the customer received an error "module/data not found". But, my PyFlink application (also mentioned to AWS numerous times) doesn't generate any errors. Then the support engineer says that UDAF should support boto3. I'm not sure if it works because AWS star principle says it doesn't work.

Danny Cranmer

07/29/2023, 6:54 AM

This is a KDA question, not a Flink question. This is not the right forum to ask vendor specific questions. I will DM you

Jeff Levesque

07/29/2023, 11:11 PM

Just for completeness, for others. I'm well aware of

my_deps/

directory structure. In fact, earlier last year I configured my CI/CD to construct the project directory and bundle into zipfile to s3. Last year, when I didn't have the correct directory structure, the pyflink application errored (maybe stopped running). I am also well aware of potential other sinks, like s3. I'm curious about SQS sinks maybe. The reason why I didn't choose to use a sink, was that it adds latency between sinks, then reading it as a source for the downstream compute (whatever choice), just to send an SNS notification, it also costs more too. So, I've always wanted to send SNS notification directly from the UDF/UDAF, and not have to sink, source, and compute it elsewhere again

Jeff Levesque

07/29/2023, 11:21 PM

requirements.txt

pretty short and sweet:

Copy code

boto3
fsspec
s3fs

The pyflink application that's zipped up and sent to s3 has the following directory structure:

Copy code

(base) jeff1evesque@computer CrappyCodeBase % ls -l
total 83328
-rw-rw-r--@  1 jeff1evesque  dungpoop      1007 Jul 28 01:22 application_properties.json
-rw-r--r--@  1 jeff1evesque  dungpoop  42646615 May 11  2022 flink-sql-connector-kinesis_2.12-1.13.2.jar
drwxr-xr-x@ 63 jeff1evesque  dungpoop      2016 Jul 29 18:59 my_deps
-rw-rw-r--@  1 jeff1evesque  dungpoop     11048 Jun  8 10:14 sliding_window.py
(base) jeff1evesque@computer CrappyCodeBase % 
(base) jeff1evesque@computer CrappyCodeBase % 
(base) jeff1evesque@computer CrappyCodeBase % cd my_deps
(base) jeff1evesque@computer my_deps % 
(base) jeff1evesque@computer my_deps % 
(base) jeff1evesque@computer my_deps % ls -l
total 312
drwxr-xr-x@  4 jeff1evesque  dungpoop     128 Jul 28 01:23 __pycache__
drwxr-xr-x@ 30 jeff1evesque  dungpoop     960 Jul 28 01:23 aiobotocore
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:23 aiobotocore-2.5.2.dist-info
drwxr-xr-x@ 62 jeff1evesque  dungpoop    1984 Jul 28 01:23 aiohttp
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:23 aiohttp-3.8.5.dist-info
drwxr-xr-x@ 13 jeff1evesque  dungpoop     416 Jul 28 01:23 aioitertools
drwxr-xr-x@  7 jeff1evesque  dungpoop     224 Jul 28 01:23 aioitertools-0.11.0.dist-info
drwxr-xr-x@  6 jeff1evesque  dungpoop     192 Jul 28 01:22 aiosignal
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:22 aiosignal-1.3.1.dist-info
-rw-rw-r--@  1 jeff1evesque  dungpoop     996 Jun  8 10:14 application_properties.py
drwxr-xr-x@  5 jeff1evesque  dungpoop     160 Jul 28 01:22 async_timeout
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul 28 01:22 async_timeout-4.0.2.dist-info
drwxr-xr-x@ 26 jeff1evesque  dungpoop     832 Jul 28 01:22 attr
drwxr-xr-x@ 11 jeff1evesque  dungpoop     352 Jul 28 01:22 attrs
drwxr-xr-x@  7 jeff1evesque  dungpoop     224 Jul 28 01:22 attrs-23.1.0.dist-info
drwxr-xr-x@  5 jeff1evesque  dungpoop     160 Jul  9 14:43 bin
drwxr-xr-x@ 15 jeff1evesque  dungpoop     480 Jul 28 01:22 boto3
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul 28 01:22 boto3-1.28.13.dist-info
drwxr-xr-x@ 48 jeff1evesque  dungpoop    1536 Jul 28 01:22 botocore
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul 28 01:22 botocore-1.31.13.dist-info
drwxr-xr-x@ 17 jeff1evesque  dungpoop     544 Jul 28 01:23 charset_normalizer
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul 28 01:23 charset_normalizer-3.2.0.dist-info
drwxr-xr-x@ 13 jeff1evesque  dungpoop     416 Jul  9 14:44 dateutil
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:22 frozenlist
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:22 frozenlist-1.4.0.dist-info
drwxr-xr-x@ 26 jeff1evesque  dungpoop     832 Jul 28 01:22 fsspec
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:22 fsspec-2023.6.0.dist-info
drwxr-xr-x@ 12 jeff1evesque  dungpoop     384 Jul 28 01:22 idna
drwxr-xr-x@  7 jeff1evesque  dungpoop     224 Jul 28 01:22 idna-3.4.dist-info
drwxr-xr-x@ 11 jeff1evesque  dungpoop     352 Jul 28 01:22 jmespath
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:22 jmespath-1.0.1.dist-info
drwxr-xr-x@ 11 jeff1evesque  dungpoop     352 Jul 28 01:22 multidict
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:22 multidict-6.0.4.dist-info
drwxr-xr-x@ 39 jeff1evesque  dungpoop    1248 Jul  9 14:44 numpy
drwxr-xr-x@ 10 jeff1evesque  dungpoop     320 Jul  9 14:43 numpy-1.24.4.dist-info
drwxr-xr-x@  5 jeff1evesque  dungpoop     160 Jul  9 14:43 numpy.libs
drwxr-xr-x@ 20 jeff1evesque  dungpoop     640 Jul  9 14:44 pandas
drwxr-xr-x@ 10 jeff1evesque  dungpoop     320 Jul  9 14:44 pandas-2.0.3.dist-info
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul  9 14:43 python_dateutil-2.8.2.dist-info
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul  9 14:44 pytz
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul  9 14:43 pytz-2023.3.dist-info
-rw-rw-r--@  1 jeff1evesque  dungpoop      18 Jun  8 10:14 requirements.txt
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul 28 01:23 s3fs
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:23 s3fs-2023.6.0.dist-info
drwxr-xr-x@ 19 jeff1evesque  dungpoop     608 Jul 28 01:22 s3transfer
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul 28 01:22 s3transfer-0.6.1.dist-info
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul  9 14:43 six-1.16.0.dist-info
-rw-r--r--@  1 jeff1evesque  dungpoop   34549 Jul  9 14:43 six.py
drwxr-xr-x@  4 jeff1evesque  dungpoop     128 Jul 28 01:19 sql
drwxr-xr-x@  7 jeff1evesque  dungpoop     224 Jul 28 01:23 typing_extensions-4.7.1.dist-info
-rw-r--r--@  1 jeff1evesque  dungpoop  111082 Jul 28 01:23 typing_extensions.py
drwxr-xr-x@  5 jeff1evesque  dungpoop     160 Jul  9 14:44 tzdata
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul  9 14:43 tzdata-2023.3.dist-info
drwxr-xr-x@  3 jeff1evesque  dungpoop      96 Jul 28 01:19 udaf
drwxr-xr-x@ 17 jeff1evesque  dungpoop     544 Jul 28 01:22 urllib3
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:22 urllib3-1.26.16.dist-info
drwxr-xr-x@  9 jeff1evesque  dungpoop     288 Jul 28 01:23 wrapt
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:23 wrapt-1.15.0.dist-info
drwxr-xr-x@ 12 jeff1evesque  dungpoop     384 Jul 28 01:22 yarl
drwxr-xr-x@  8 jeff1evesque  dungpoop     256 Jul 28 01:22 yarl-1.9.2.dist-info
(base) jeff1evesque@computer my_deps %

Open in Slack

Previous Next