Show an example of how to implement a custom audio...
# ask-ai
p
Show an example of how to implement a custom audio processors to control input gain
t
Thanks for reaching out, Bilal. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Here are examples of how to implement a custom audio processor to control input gain in LiveKit, tailored to different platforms: Android Implementation:
Copy code
class GainAudioProcessor : AudioProcessorInterface {
    private var gain: Float = 1.0f

    override fun initializeAudioProcessing(sampleRateHz: Int, numChannels: Int) {
        // Initialize your processor with sample rate and channel info
    }

    override fun processAudio(numBands: Int, numFrames: Int, buffer: ByteBuffer) {
        // Apply gain control to the audio buffer
        // Convert buffer to audio samples, apply gain, and write back
    }

    override fun resetAudioProcessing(newRate: Int) {
        // Handle sample rate changes
    }

    fun setGain(newGain: Float) {
        gain = newGain
    }
}

val audioProcessorOptions = AudioProcessorOptions(
    capturePostProcessor = GainAudioProcessor()
)

val liveKitOverrides = LiveKitOverrides(
    audioOptions = AudioOptions(
        audioProcessorOptions = audioProcessorOptions
    )
)
iOS/Swift Implementation:
Copy code
class GainAudioProcessor: AudioCustomProcessingDelegate {
    var gain: Float = 1.0

    func audioProcessingInitialize(sampleRate sampleRateHz: Int, channels: Int) {
        // Initialize with sample rate and channel count
    }

    func audioProcessingProcess(audioBuffer: LKAudioBuffer) {
        // Apply gain control to the audio buffer
        // Modify audioBuffer.data in place
    }

    func audioProcessingRelease() {
        // Clean up resources
    }
}

AudioManager.shared.capturePostProcessingDelegate = GainAudioProcessor()
Python Implementation:
Copy code
from livekit.rtc import AudioProcessingModule

# Enable auto gain control
apm = AudioProcessingModule(
    auto_gain_control=True,
    echo_cancellation=False,
    noise_suppression=False,
    high_pass_filter=False
)

# Process audio frames
apm.process_stream(audio_frame)
Agent-Side Volume Control:
Copy code
def _adjust_volume_in_frame(self, frame: rtc.AudioFrame, gain: float) -> rtc.AudioFrame:
    audio_data = np.frombuffer(frame.data, dtype=np.int16)
    audio_float = audio_data.astype(np.float32) / np.iinfo(np.int16).max
    audio_float = audio_float * gain  # Apply gain
    processed = (audio_float * np.iinfo(np.int16).max).astype(np.int16)

    return rtc.AudioFrame(
        data=processed.tobytes(),
        sample_rate=frame.sample_rate,
        num_channels=frame.num_channels,
        samples_per_channel=len(processed) // frame.num_channels,
    )
Key Considerations: • Use
capturePostProcessor
for input gain control. • Handle the specific audio format (PCM 16-bit is common). • Ensure real-time performance to avoid audio dropouts. • Adapt to platform-specific APIs while following similar buffer processing patterns. Let me know if you need further details on any specific platform or implementation! Sources: livekit-android-sdk/src/main/java/io/livekit/android/audio/AudioProcessingC... | Sources/LiveKit/Audio/MixerEngineObserver.swift | livekit-rtc/livekit/rtc/apm.py | Agent speech and audio | LiveKit Docs
p
Show me Javascript example
t
Hey there! Let me walk you through a super cool way to control your audio input gain in LiveKit using JavaScript. It's easier than you might think! 😊 Custom Input Gain Processor:
Copy code
class InputGainProcessor {
  constructor(gainValue = 1.0) {
    this.name = 'InputGainProcessor';
    this.gainValue = gainValue;
    this.audioContext = null;
    this.sourceNode = null;
    this.gainNode = null;
    this.destinationNode = null;
    this.processedTrack = null;
  }

  async init(processorOptions) {
    const { track, audioContext } = processorOptions;
    this.audioContext = audioContext;

    // Create audio processing chain
    const stream = new MediaStream([track]);
    this.sourceNode = audioContext.createMediaStreamSource(stream);

    // Create gain node for volume control
    this.gainNode = audioContext.createGain();
    this.gainNode.gain.setTargetAtTime(this.gainValue, 0, 0.1);

    // Create destination for processed audio
    this.destinationNode = audioContext.createMediaStreamDestination();

    // Connect the processing chain
    this.sourceNode.connect(this.gainNode);
    this.gainNode.connect(this.destinationNode);

    // Get the processed track
    this.processedTrack = this.destinationNode.stream.getAudioTracks()[0];
  }

  setGain(gainValue) {
    this.gainValue = gainValue;
    if (this.gainNode) {
      this.gainNode.gain.setTargetAtTime(gainValue, 0, 0.1);
    }
  }

  destroy() {
    if (this.sourceNode) {
      this.sourceNode.disconnect();
    }
    if (this.gainNode) {
      this.gainNode.disconnect();
    }
  }
}
Using the Processor with LocalAudioTrack:
Copy code
// Create audio context (required for processors)
const audioContext = new (window.AudioContext || window.webkitAudioContext)();

// Create local audio track with audio context
const localAudioTrack = await LocalAudioTrack.create({
  // audio capture options
}, audioContext);

// Create and apply the gain processor
const gainProcessor = new InputGainProcessor(0.5); // 50% gain
await localAudioTrack.setProcessor(gainProcessor);

// Control gain dynamically
gainProcessor.setGain(0.8); // 80% gain
Key Notes: • An
AudioContext
must be set on the
LocalAudioTrack
for processors to work. • The processor should implement an
init
method that receives
processorOptions
containing the track and audioContext. • The processed audio should be available via a
processedTrack
property. I hope this helps you get started with audio processing in LiveKit! It's a bit technical, but once you get the hang of it, you'll be tweaking audio like a pro. If you're scratching your head or need more guidance, I'm totally here to help – just give me a shout! 🎧🚀 Sources: quickstart/src/miclevel.js | src/room/track/LocalAudioTrack.ts
p
When publishing the track, should I publish the
const localAudioTrack = await LocalAudioTrack.create({ // audio capture options }, audioContext);
or the
gainProcessor.processedTrack
?
t
Hey there! Let me break down something important about publishing tracks in LiveKit with a custom audio processor. You'll want to publish the
LocalAudioTrack
, not the
gainProcessor.processedTrack
directly. Trust me, it makes things way smoother! Here's the scoop: • When you set a processor on a
LocalAudioTrack
using
setProcessor()
, LiveKit's SDK is super smart and automatically handles all the technical integration for you. If your processor has a
processedTrack
, the SDK will seamlessly replace the track on the sender with the processed version. • The
LocalAudioTrack
interface stays exactly the same, and the SDK works its magic to make sure your processed audio reaches other participants. • You can keep working with your
LocalAudioTrack
object just like normal, while the SDK does all the heavy lifting behind the scenes. Bottom line: Stick with publishing the
LocalAudioTrack
, and the SDK will take care of sending that processed audio to everyone else. Easy peasy! Sources: lib/src/track/options.dart | src/room/participant/LocalParticipant.ts