I ended up with this then

begin
  require 'voxpupuli/acceptance/rake'
rescue LoadError
  # only available if gem group acceptance is installed
else
  # We use fixtures in our modules, which is not the default
  task beaker: 'fixtures:prep'
end

Rake::Task[:beaker].enhance do
  Rake::Task['fixtures:clean'].invoke
end

And it seems to clean up nicely 🙂 Thanks @bastelfreak