purple-rainbow-1246
07/25/2025, 3:45 PMtall-airline-16756
07/25/2025, 3:45 PMtall-airline-16756
07/25/2025, 3:46 PMclass GainAudioProcessor : AudioProcessorInterface {
private var gain: Float = 1.0f
override fun initializeAudioProcessing(sampleRateHz: Int, numChannels: Int) {
// Initialize your processor with sample rate and channel info
}
override fun processAudio(numBands: Int, numFrames: Int, buffer: ByteBuffer) {
// Apply gain control to the audio buffer
// Convert buffer to audio samples, apply gain, and write back
}
override fun resetAudioProcessing(newRate: Int) {
// Handle sample rate changes
}
fun setGain(newGain: Float) {
gain = newGain
}
}
val audioProcessorOptions = AudioProcessorOptions(
capturePostProcessor = GainAudioProcessor()
)
val liveKitOverrides = LiveKitOverrides(
audioOptions = AudioOptions(
audioProcessorOptions = audioProcessorOptions
)
)
iOS/Swift Implementation:
class GainAudioProcessor: AudioCustomProcessingDelegate {
var gain: Float = 1.0
func audioProcessingInitialize(sampleRate sampleRateHz: Int, channels: Int) {
// Initialize with sample rate and channel count
}
func audioProcessingProcess(audioBuffer: LKAudioBuffer) {
// Apply gain control to the audio buffer
// Modify audioBuffer.data in place
}
func audioProcessingRelease() {
// Clean up resources
}
}
AudioManager.shared.capturePostProcessingDelegate = GainAudioProcessor()
Python Implementation:
from livekit.rtc import AudioProcessingModule
# Enable auto gain control
apm = AudioProcessingModule(
auto_gain_control=True,
echo_cancellation=False,
noise_suppression=False,
high_pass_filter=False
)
# Process audio frames
apm.process_stream(audio_frame)
Agent-Side Volume Control:
def _adjust_volume_in_frame(self, frame: rtc.AudioFrame, gain: float) -> rtc.AudioFrame:
audio_data = np.frombuffer(frame.data, dtype=np.int16)
audio_float = audio_data.astype(np.float32) / np.iinfo(np.int16).max
audio_float = audio_float * gain # Apply gain
processed = (audio_float * np.iinfo(np.int16).max).astype(np.int16)
return rtc.AudioFrame(
data=processed.tobytes(),
sample_rate=frame.sample_rate,
num_channels=frame.num_channels,
samples_per_channel=len(processed) // frame.num_channels,
)
Key Considerations:
• Use capturePostProcessor for input gain control.
• Handle the specific audio format (PCM 16-bit is common).
• Ensure real-time performance to avoid audio dropouts.
• Adapt to platform-specific APIs while following similar buffer processing patterns.
Let me know if you need further details on any specific platform or implementation!
Sources: livekit-android-sdk/src/main/java/io/livekit/android/audio/AudioProcessingC... | Sources/LiveKit/Audio/MixerEngineObserver.swift | livekit-rtc/livekit/rtc/apm.py | Agent speech and audio | LiveKit Docspurple-rainbow-1246
07/25/2025, 3:47 PMtall-airline-16756
07/25/2025, 3:48 PMclass InputGainProcessor {
constructor(gainValue = 1.0) {
this.name = 'InputGainProcessor';
this.gainValue = gainValue;
this.audioContext = null;
this.sourceNode = null;
this.gainNode = null;
this.destinationNode = null;
this.processedTrack = null;
}
async init(processorOptions) {
const { track, audioContext } = processorOptions;
this.audioContext = audioContext;
// Create audio processing chain
const stream = new MediaStream([track]);
this.sourceNode = audioContext.createMediaStreamSource(stream);
// Create gain node for volume control
this.gainNode = audioContext.createGain();
this.gainNode.gain.setTargetAtTime(this.gainValue, 0, 0.1);
// Create destination for processed audio
this.destinationNode = audioContext.createMediaStreamDestination();
// Connect the processing chain
this.sourceNode.connect(this.gainNode);
this.gainNode.connect(this.destinationNode);
// Get the processed track
this.processedTrack = this.destinationNode.stream.getAudioTracks()[0];
}
setGain(gainValue) {
this.gainValue = gainValue;
if (this.gainNode) {
this.gainNode.gain.setTargetAtTime(gainValue, 0, 0.1);
}
}
destroy() {
if (this.sourceNode) {
this.sourceNode.disconnect();
}
if (this.gainNode) {
this.gainNode.disconnect();
}
}
}
Using the Processor with LocalAudioTrack:
// Create audio context (required for processors)
const audioContext = new (window.AudioContext || window.webkitAudioContext)();
// Create local audio track with audio context
const localAudioTrack = await LocalAudioTrack.create({
// audio capture options
}, audioContext);
// Create and apply the gain processor
const gainProcessor = new InputGainProcessor(0.5); // 50% gain
await localAudioTrack.setProcessor(gainProcessor);
// Control gain dynamically
gainProcessor.setGain(0.8); // 80% gain
Key Notes:
• An AudioContext must be set on the LocalAudioTrack for processors to work.
• The processor should implement an init method that receives processorOptions containing the track and audioContext.
• The processed audio should be available via a processedTrack property.
I hope this helps you get started with audio processing in LiveKit! It's a bit technical, but once you get the hang of it, you'll be tweaking audio like a pro. If you're scratching your head or need more guidance, I'm totally here to help – just give me a shout! 🎧🚀
Sources: quickstart/src/miclevel.js | src/room/track/LocalAudioTrack.tspurple-rainbow-1246
07/25/2025, 3:55 PMconst localAudioTrack = await LocalAudioTrack.create({ // audio capture options }, audioContext); or the gainProcessor.processedTrack?tall-airline-16756
07/25/2025, 3:56 PMLocalAudioTrack, not the gainProcessor.processedTrack directly. Trust me, it makes things way smoother!
Here's the scoop:
• When you set a processor on a LocalAudioTrack using setProcessor(), LiveKit's SDK is super smart and automatically handles all the technical integration for you. If your processor has a processedTrack, the SDK will seamlessly replace the track on the sender with the processed version.
• The LocalAudioTrack interface stays exactly the same, and the SDK works its magic to make sure your processed audio reaches other participants.
• You can keep working with your LocalAudioTrack object just like normal, while the SDK does all the heavy lifting behind the scenes.
Bottom line: Stick with publishing the LocalAudioTrack, and the SDK will take care of sending that processed audio to everyone else. Easy peasy!
Sources: lib/src/track/options.dart | src/room/participant/LocalParticipant.ts