I realized that the log I provided above was the live-kit server log... but that doesn't really give the whole picture. The log from the egress server would be useful as well since the ability to buffer/retry to overcome the flapping of the S3 connection would need to be coming from there. I've recreated the scenario and captured the relevant sections of the livekit-server, egress and testcase logs and attached.

$ livekit-cli list-egress
Using default project STAGING
+-----------------+---------------+--------------------------------+-----------------------------------------------------------------------------------------+
|    EGRESSID     |    STATUS     |           STARTED AT           |                                          ERROR                                          |
+-----------------+---------------+--------------------------------+-----------------------------------------------------------------------------------------+
| EG_GARdcTmB7W5U | EGRESS_FAILED | 2023-10-09 14:41:07.72940668   | S3 upload failed: RequestError: send request failed caused by: Put                      |
|                 |               | -0700 PDT                      | "<http://zoom-live.nuobject.io/livekitbucket/3b9cb3dd7aab0bfdae31b15cbf6aeebc_00011.ts>": |
|                 |               |                                | readfrom tcp 10.9.174.213:48460->10.9.172.19:8501: write tcp                            |
|                 |               |                                | 10.9.174.213:48460->10.9.172.19:8501: write: connection reset by peer                   |
+-----------------+---------------+--------------------------------+-----------------------------------------------------------------------------------------+


LiveKit RoomId:  3b9cb3dd7aab0bfdae31b15cbf6aeebc

Hi @able-gigabyte-21598. We've been trying different approaches to making this more resilient for our environment and I've just tried out a change that lets our "flapping" resilency test (described above) work. The change was to add a more robust retry policy via environment vars and required some code changes in s3.go. It looks like this

//get the content of the MAX_S3_RETRIES environment variable
	//if it is set, use it as the max number of retries after conversion to int as part of custom retryer
	//if it is not set, use the maxRetries constant as set above and don't define a custom retryer
	maxRetriesEnv, exists := os.LookupEnv("MAX_S3_RETRIES")
	if exists {
		_maxRetries, err := strconv.Atoi(maxRetriesEnv)
		if err != nil {
			logger.Warnw("MAX_S3_RETRIES environment variable is not an integer", err, maxRetriesEnv)
		} else {
			maxDelayEnv, exists := os.LookupEnv("MAX_S3_DELAY")
			if exists {
				_maxDelay, err := strconv.Atoi(maxDelayEnv)
				if err != nil {
					logger.Warnw("MAX_S3_DELAY environment variable is not an integer", err, maxDelayEnv)
				} else {
					awsConfig.Retryer = &client.DefaultRetryer{
						NumMaxRetries:    _maxRetries,
						MaxRetryDelay:    time.Duration(_maxDelay) * time.Second,
						MinThrottleDelay: time.Duration(_maxDelay) * time.Second,
					}
					logger.Infow("Configuring s3 with retryer", "maxRetries", _maxRetries,
						"maxDelaySecs", _maxDelay)
				}
			}
		}
	}

It might be a little heavy handed, but we're testing with these settings

- name: MAX_S3_RETRIES
              value: '90'
            - name: MAX_S3_DELAY
              value: '30'

We might need to look at adjusting and also the amount of temp storage we use for the segments in the pods... but we're making progress. We've got this mod and a mod to establish an HTTP proxy for the S3 connection. I think we'd probably want to move both out of ENV vars and into the S3 configuration at some point. Maybe a free-form map of additional parms wold be a nice thing to have in that config to make it easier to add tweaks in the future.