If i do not have access to Distributed File System or cloud's s3 or gcs, can i create persistent volumes in k8s and manage my state there ? has any one done that for their production jobs . any thoughts ?

You can share the same volume to all your pods, haven’t done it in prod though.