I'm passively looking at a release trying to sus out issues early. Hoping to do one next week.

any reason why we're not auto-releasing on every merge to main? maybe via tools like: https://python-semantic-release.readthedocs.io/en/latest/ (only used the Node and Go equivalents myself)

It's a fast moving project. Things break regularly. If anyone wants to always have the latest, they can just use

main

tag.

setting up semantic release correctly won't allow main to be broken

My apologies. Maybe "fast" isn't the right word here. Curious how the semantic release won't allow main to be broken? Can you elaborate on that?

tests won't pass

Hey @polite-architect-52194 if we had good enough e2e tests that might be true but we don't, so @better-park-34863 has to do some manual testing on every release. It's unfortunately not trivial to build tests for everything, esp when LLMs are involved

Yeah I would love to have tests that cover all the core functionality. 🙂 Until then main can be broken.

@nutritious-balloon-65098 where are we with respect to the above cited recommendations?

I'm on it! Graham Neubig can track my progress here.

Hth 😃

• Integration tests run nightly/on-demand rather than blocking PRs is this a cost concern? (I imagine these tests use lots of tokens?) if yes, then this is a business decision, and not a technical one. I.e.: to what extent can you accept broken main, and the reputational risk that brings my take is main should never be broken, i.e.: all integration and e2e tests must have passed

Integration tests take too long to run and they block us from merging PRs quickly

https://docs.gitlab.com/ci/pipelines/merge_trains/ don't know the GitHub equivalent to this

I think we could potentially ask OpenHands to QA OpenHands, it should be possible

@nutritious-balloon-65098 if we compare against the elite cohort, where does OpenHands sit? what would be top 5 DORA capabilities to work on to improve our situation?

Paul Cuciureanu is not authorized to send messages to this conversation.

Hi @limited-student-10747, I had a stab with o3 and deep research at this topic, against the OpenHands codebase https://chatgpt.com/share/6873c90e-7a2c-8002-bcaa-5e2c688a3d37 Should I create GitHub issues, so OpenHands could action some of these? I don't think I saw existing ones tackling these concerns Below are five [...] moves that will push OpenHands toward DORA-elite performance. Each item includes the concrete first step, the primary DORA capability it improves, and an evidence link from the current repo. 1. Ship a deterministic, full-stack E2E suite • _Why_: Existing integration runs are scheduled or label-gated, leaving gaps and flakiness; a sealed end-to-end flow catches regressions earlier. • _First-day move_: Fork the

integration-runner.yml

, spin up a

docker-compose

profile that includes a stub LLM container and Playwright scripts that drive the React UI against the backend. Seed fixed test data so failures are deterministic. • _DORA boost_: Test Automation → Fast Feedback. 2. Guard

main

with a merge queue and staged CI • _Why_: PRs can merge while deeper tests are still red; a queue ensures code is tested in the exact merge order. • _First-day move_: Turn on GitHub “Merge Queue”, mark lint, unit, and the new E2E smoke job as required branch-protection checks. Heavy E2E runs once per batch, so latency stays low. • _DORA boost_: Continuous Integration + Trunk-Based Development. 3. Automate releases and feature flags • _Why_: Weekly manual releases slow feedback; flags make risky code safe to ship daily. • _First-day move_: Add

python-semantic-release

with two branches—`main` (stable) and

next

(prerelease). Create

feature_flags.toml

/

featureFlags.ts

and migrate the existing

VSCODE_IN_NEW_TAB

env var into it; default all new flags to off. • _DORA boost_: Continuous Delivery + Deployment Automation. 4. Add agent-aware observability and smoke deploys • _Why_: Bugs that surface only in runtime agent loops are invisible until users complain. • _First-day move_: New CI job builds the Docker image, launches a one-shot staging stack, runs a 60-second canned agent task, emits OpenTelemetry spans, and tears down. Fail the job and alert Slack if the task or startup fails. • _DORA boost_: Monitoring & Proactive Failure Notification. 5. Expose flow and community metrics • _Why_: Without transparency you can’t improve PR close rate, lead time, or bus factor. • _First-day move_: Nightly GitHub Action calls the GraphQL API, records PR close rate, median lead time, time-to-first-response, bus factor, and release frequency, then pushes JSON to

docs/metrics/

. Publish a lightweight Grafana dashboard pointing at those files. • _DORA boost_: Learning Culture + Generative Org Culture. Uncertainty: exact file paths and job names may differ; adjust commands to match repo layout.