Hi all, we are experiencing a problem that messages in

sync.search.product

queue are not being processed (retried all over again). I was looking at the errors logs and couldn’t find anything specific for this queue. Only thing i found is the following:

bash-5.0# tail -f /var/log/spryker/php_errors.log
[13-Aug-2020 09:09:19 UTC] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /data/vendor/spryker/util-encoding/src/Spryker/Service/UtilEncoding/Model/Json.php on line 55
[13-Aug-2020 09:09:19 UTC] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /data/vendor/spryker/error-handler/src/Spryker/Shared/ErrorHandler/ErrorHandlerEnvironment.php on line 93

Those errors are being thrown in jenkins container every 5-7 seconds. Did anyone experience these issues and have a suggestion how to fix it?

Would it be that message in sync queue is too big? Probably reducing BULK size would help.

last time I had similar error there were messages that could not be processed. What happens is that when there is an error the worker tries to process messages individually and when it runs out of memory it fails and messages are not acknowledged. Reducing the batch size helps and Mike suggested.

as i can see default chunk size is set to 10000. What should be the reduced value in this case?

Chunk value out of the box is 500, I have no idea where 10000 is coming from.

500 might be for publishing, 10000 for syncing, it sounds fine. The elastic search document can be quite big, depending on your data. Putting the whole batch into memory can be troublesome. So yes, as a quick solution - try to reduce the chunk, but I would also check - aren’t you sending too much data to the elastic search?

More attributes you have the bigger data you need to pump. (for products)

I will try reducing the chunk size at first and then investigate further.

That’s probably a root cause.

I reduced the chunk size to 500 and it is working now.

I meant 128 Mb for PHP used memory. Not the data itself.