Patent 11381906 - Conference system with a microphone array system and a method of speech acquisition in a conference system > Description
The present application is a continuation of U.S. patent application Ser. No. 15/780,787 filed on Jun. 1, 2018, which claims priority from International Patent Application No. PCT/EP2016/079720 filed on Dec. 5, 2016, which claims priority from U.S. patent application Ser. No. 14/959,387 filed on Dec. 4, 2015, the disclosures of which are incorporated herein by reference in their entirety.
It is noted that citation or identification of any document in this application is not an admission that such document is available as prior art to the present invention.
The invention relates to a conference system as well as a method of speech acquisition in a conference system.
In a conference system, the speech signal of one or more participants, typically located in a conference room, must be acquired such that it can be transmitted to remote participants or for local replay, recording or other processing.
Each microphone 1100 may have a suitable directivity pattern, e.g. cardioid and is directed to the mouth of the corresponding participant 1010. This arrangement enables predominant acquisition of the participants' 1010 speech and reduced acquisition of disturbing noise. The microphone signals from the different participants 1010 may be summed together and can be transmitted to remote participants. A disadvantage of this solution is the microphone 1100 requiring space on the table 1020, thereby restricting the participants work space. Furthermore for proper speech acquisition the participants 1010 have to stay at their seat. If a participant 1010 walks around in the room 1001, e.g. for using a whiteboard for additional explanation, this arrangement leads to degraded speech acquisition results.
US 2008/0247567 A1 shows a two-dimensional microphone array for creating an audio beam pointing to a given direction.
U.S. Pat. No. 6,731,334 B1 shows a microphone array used for tracking the position of a speaking person for steering a camera.
It's an object of the invention to provide a conference system that enables enhanced freedom of the participants at improved speech acquisition and reduced setup effort.
According to the invention, a conference system is provided which comprises a microphone array unit having a plurality of microphone capsules arranged in or on a board mountable on or in a ceiling of a conference room. The microphone array unit has a steerable beam and a maximum detection angle range. A processing unit is configured to receive the output signals of the microphone capsules and to steer the beam based on the received output signal of the microphone array unit. The processing unit is also configured to control the microphone array to limit the detection angle range to exclude at least one predetermined exclusion sector in which noise is located.
The invention also relates to a conference system having a microphone array unit having a plurality of microphone capsules arranged in or on a board mountable on or in a ceiling of a conference room. The microphone array unit has a steerable beam. A processing unit is provided which is configured to detect a position of an audio source based on the output signals of the microphone array unit. The processing unit comprises a direction recognition unit which is configured to identify a direction of an audio source and to output a direction signal. The processing unit comprises filters for each microphone signal, delay units configured to individually add an addressable delay to the output of the filters, a summing unit configured to sum the outputs of the delay units and a frequency response correction filter configured to receive the output of the summing unit and to output an overall output signal to the processing unit. The processing unit also comprises a delay control unit configured to receive the direction signal and to convert directional information into delay values for the delay units. The delay units are configured to receive those delay values and to adjust their delay time accordingly.
According to an aspect of the invention, the processing unit comprises a correction control unit configured to receive the direction signal from the direction recognition unit and to convert the direction information into a correction control signal which is used to adjust the frequency response correction filter. The frequency response correction filter can be performed as an adjustable equalizing wherein the equalizing is adjusted based on the dependency of the frequency response of the audio source to the direction of the audio beam. The frequency response correction filter is configured to compensate deviations from a desired amplitude frequency response by a filter having an inverted amplitude frequency response.
The invention also relates to a microphone array unit having a plurality of microphone capsules arranged in or on a board mountable in or on a ceiling in a conference room. The microphone array unit has a steerable beam and a maximum detection angle. The microphone capsules are arranged on one side of the board in close distance to the surface wherein the microphone capsules are arranged in connection lines from a corner of the board to the center of the board. Starting at the center, the distance between two neighboring microphone capsules along the connection line is increasing with increasing distance from the center.
The present invention also relates to a conference system having a microphone array unit having a plurality of microphone capsules arranged in or on a board mountable on or in a ceiling of a conference room. The microphone array unit has a steerable beam. The processing unit is configured to detect a position of an audio source based on the output signals of the microphone capsules. The processing unit comprises filters for each microphone signal delay units configured to individually add an adjustable delay to the output of the filter's summing unit configured to sum the outputs of the delay units and a frequency response correction filter configured to receive the output of the summing unit and to output an overall output signal of the processing unit. The processing unit comprises a direction recognition unit which is configured to identify a direction of an audio source based on a Steered Response Power with Phase Transformation (SRP-PRAT) algorithm and to output a direction signal. By successively repeating the summation of the outputs of the delay units over several points in space as part of a predefined search grid, a SRP-PHAT score is determined by the direction recognition unit for each point in space. The position of the highest SRP-PRAT score is considered as a position of an audio source sound. If a block of signals achieves a SRP-PHAT score of less than a threshold, the beam can be kept at a last valid position to give a maximum SRP-PRAT score above the threshold.
It is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clear understanding of the present invention, while eliminating, for purposes of clarity, many other elements which are conventional in this art. Those of ordinary skill in the art will recognize that other elements are desirable for implementing the present invention. However, because such elements are well known in the art, and because they do not facilitate a better understanding of the present invention, a discussion of such elements is not provided herein.
The present invention will now be described in detail on the basis of exemplary embodiments.
The audio signals acquired by the microphone capsules 2001-2004 are fed to a processing unit 2400 of the microphone array unit 2000. Based on the output signals of the microphone capsules, the processing unit 2400 identifies the direction (a spherical angle relating to the microphone array; this may include a polar angle and an azimuth angle; optionally a radial distance) in which a speaking person is located. The processing unit 2400 then executes an audio beam 2000b forming based on the microphone capsule signals for predominantly acquiring sound coming from the direction as identified.
The speaking person direction can periodically be re-identified and the microphone beam direction 2000b can be continuously adjusted accordingly. The whole system can be preinstalled in a conference room and preconfigured so that no certain setup procedure is needed at the start of a conference for preparing the speech acquisition. At the same time the speaking person tracing enables a predominant acquisition of the participants' speech and reduced acquisition of disturbing noise. Furthermore the space on the table remains free and the participants can walk around in the room at remaining speech acquisition quality.
The carrier board 2020 can optionally have a square shape. Preferably it is mounted to the ceiling in a conference room in a way that the surface is arranged in a horizontal orientation. On the surface directing down from the ceiling the microphone capsules are arranged.
Here, the capsules are arranged on the diagonals of the square shape. There are four connection lines 2020a-2020d, each starting at the middle point of the square and ending at one of the four edges of the square. Along each of those four lines 2020a-2020d a number of microphone capsules 2001-2017 is arranged in a common distance pattern. Starting at the middle point the distance between two neighboring capsules along the line is increasing with increasing distance from the middle point. Preferably, the distance pattern represents a logarithmic function with the distance to the middle point as argument and the distance between two neighboring capsules as function value. Optionally a number of microphones which are placed close to the center have an equidistant linear spacing, resulting in an overall linear-logarithmic distribution of microphone capsules.
The outermost capsule (close to the edge) 2001, 2008, 2016, 2012 on each connection line still keeps a distance to the edge of the square shape (at least the same distance as the distance between the two innermost capsules). This enables the carrier board to also block away reflected sound from the outermost capsules and reduces artifacts due to edge diffraction if the carrier board is not flush mounted into the ceiling.
Optionally the microphone array further comprises a cover for covering the microphone surface side of the carrier board and the microphone capsules. The cover preferably is designed to be acoustically transparent, so that the cover does not have a substantial impact on the sound reaching the microphone capsules.
Preferably all microphone capsules are of the same type, so that they feature the same frequency response and the same directivity pattern. The preferred directivity pattern for the microphone capsules 2001-2017 is omnidirectional as this provides as close as possible a sound incident angle independent frequency response for the individual microphone capsules. However, other directivity patterns are possible.
Specifically cardioid pattern microphone capsules can be used to achieve better directivity, especially at low frequencies. The capsules are preferably arranged mechanically parallel to each other in the sense that the directivity pattern of the capsules all point into the same direction. This is advantageous as it enables the same frequency response for all capsules at a given sound incidence direction, especially with respect to the phase response.
In situations where the microphone system is not flush mounted in the ceiling, further optional designs are possible.
The processing unit 2400 furthermore comprises individual filters 2421-2424 for each microphone signal. The output of each individual filters 2421-2424 is fed to an individual delay unit 2431-2434 for individually adding an adjustable delay to each of those signals. The outputs of all those delay units 2431-2434 are summed together in a summing unit 2450. The output of the summing unit 2450 is fed to a frequency response correction filter 2460. The output signal of the frequency response correction filter 2460 represents the overall output signal 2470 of the processing unit 2400. This is the signal representing a speaking person's voice signal coming from the identified direction.
Directing the audio beam to the direction as identified by the direction recognition unit 2440 in the embodiment of
The processing unit 2400 furthermore comprises a correction control unit 2443. The correction control unit 2443 receives the direction information 2441 from the direction recognition unit 2440 and converts it into a correction control signal 2444. The correction control signal 2444 is used to adjust the frequency response correction filter 2460. The frequency response correction filter 2460 can be performed as an adjustable equalizing unit. The setting of this equalizing unit is based on the finding that the frequency response as observed from the speaking person's voice signal to the output of the summing unit 2450 is dependent to the direction the audio beam 2000b is directed to. Therefore the frequency response correction filter 2460 is configured to compensate deviations from a desired amplitude frequency response by a filter 2460 having an inverted amplitude frequency response.
The position or direction recognition unit 2440 detects the position of audio sources by processing the digitized signals of at least two of the microphone capsules as depicted in
When a microphone array with a conventional Delay and Sum Beamformer (DSB) is successively steered at points in space by adjusting its steering delays, the output power of the Beamformer can be used as measure where a source is located. The steered response power (SRP) algorithm performs this task by calculating generalized cross correlations (GCC) between pairs of input signals and comparing them against a table of expected time difference of arrival (TDOA) values. If the signals of two microphones are practically time delayed versions of each other, which will be the case for two microphones picking up the direct path of a sound source in the far field, their GCC will have a distinctive peak at the position corresponding to the TDOA of the two signals and it will be close to zero for all other positions. SRP uses this property to calculate a score by summing the GCCs of a multitude of microphone pairs at the positions of expected TDOAs, corresponding to a certain position in space. By successively repeating this summation over several points in space that are part of a pre-defined search grid, a SRP-PHAT score is gathered for each point in space. The position with the highest SRP-PHAT score is considered as the sound source position.
Afterwards the phase transform 2521-2523 and pairwise cross-correlation of signals 2531-2533 is performed before transforming the signals into the time domain again 2541-2543. These GCCs are then fed into the scoring unit 2550. The scoring unit computes a score for each point in space on a pre-defined search grid. The position in space that achieves the highest score is considered to be the sound source position.
By using a phase transform weighting for the GCCs, the algorithm can be made more robust against reflections, diffuse noise sources and head orientation. In the frequency domain the phase transform as performed in the units 2521-2523 divides each frequency bin with its amplitude, leaving only phase information. In other words the amplitudes are set to 1 for all frequency bins.
The SRP-PHAT algorithm as described above and known from prior art has some disadvantages that are improved in the context of this invention.
In a typical SRP-PHAT scenario the signals of all microphone capsules of an array will be used as inputs to the SRP-PHAT algorithm, all possible pairs of these inputs will be used to calculate GCCs and the search grid will be densely discretizing the space around the microphone array. All this leads to very high amounts of processing power required for the SRP-PHAT algorithm.
According to an aspect of the invention, a couple of techniques are introduced to reduce the processing power needed without sacrificing for detection precision. In contrast to using the signals of all microphone capsules and all possible microphone pairs, preferably a set of microphones can be chosen as inputs to the algorithm or particular microphone pairs can be chosen to calculate GCCs of. By choosing microphone pairs that give good discrimination of points in space, the processing power can be reduced while keeping a high amount of detection precision.
As the microphone system according to the invention only requires a look direction to point to a source, it is further not desirable to discretize the whole space around the microphone array into a search grid, as distance information is not necessarily needed. If a hemisphere with a radius much larger than the distance between the microphone capsules used for the GCC pairs is used, it is possible to detect the direction of a source very precisely, while at the same time reducing the processing power significantly, as only a hemisphere search grid is to be evaluated. Furthermore the search grid is independent from room size and geometry and risk of ambiguous search grid positions e.g. if a search grid point would be located outside of the room. Therefore, this solution is also advantageous to prior art solutions to reduce the processing power like coarse to fine grid refinement, where first a coarse search grid is evaluated to find a coarse source position and afterwards the area around the detected source position will be searched with a finer grid to find the exact source position.
It can be desirable to also have distance information of the source, in order to e.g. adapt the beamwidth to the distance of the source to avoid a too narrow beam for sources close to the array or in order to adjust the output gain or EQ according to the distance of the source.
Besides of significantly reducing the required processing power of typical SRP-PHAT implementations, the robustness against disturbing noise sources has been improved by a set of measures. If there is no person speaking in the vicinity of the microphone system and the only signals picked up are noise or silence, the SRP-PHAT algorithm will either detect a noise source as source position or especially in the case of diffuse noises or silence, quasi randomly detect a “source” anywhere on the search grid. This either leads to predominant acquisition of noise or audible audio artifacts due to a beam randomly pointing at different positions in space with each block of audio. It is known from prior art that this problem can be solved to some extent by computing the input power of at least one of the microphone capsules and to only steer a beam if the input power is above a certain threshold. The disadvantage of this method is that the threshold has to be adjusted very carefully depending on the noise floor of the room and the expected input power of a speaking person. This requires interaction with the user or at least time and effort during installation. This behavior is depicted in
The invention overcomes this problem by using the SRP-PHAT score that is already computed for the source detection as a threshold metric (SRP-threshold) instead or in addition to the input power. The SRP-PHAT algorithm is insensitive to reverberation and other noise sources with a diffuse character. In addition most noise sources as e.g. air conditioning systems have a diffuse character while sources to be detected by the system usually have a strong direct or at least reflected sound path. Thus most noise sources will produce rather low SRP-PHAT scores, while a speaking person will produce much higher scores. This is mostly independent of the room and installation situation and therefore no significant installation effort and no user interaction is required, while at the same time a speaking person will be detected and diffuse noise sources will not be detected by the system. As soon as a block of input signals achieves a SRP-PHAT score of less than the threshold, the system can e.g. be muted or the beam can be kept at the last valid position that gave a maximum SRP-PHAT score above the threshold. This avoids audio artifacts and detection of unwanted noise sources. The advantage over a sound energy threshold is depicted in
Thus this gated SRP-PRAT algorithm is robust against diffuse noise sources without the need of tedious setup and/or control by the user.
However, noise sources with a non-diffuse character that are present at the same or higher sound energy level as the wanted signal of a speaking person, might still be detected by the gated SRP-PRAT algorithm. Although the phase transform will result in frequency bins with uniform gain, a source with high sound energy will still dominate the phase of the systems input signals and thus lead to predominant detection of such sources. These noise sources can for example be projectors mounted closely to the microphone system or sound reproduction devices used to play back the audio signal of a remote location in a conference scenario. Another part of the invention is to make use of the pre-defined search grid of the SRP-PRAT algorithm to avoid detection of such noise sources. If areas are excluded from the search grid, these areas are hidden for the algorithm and no SRP-PHAT score will be computed for these areas. Therefore no noise sources situated in such a hidden area can be detected by the algorithm. Especially in combination with the introduced SRP-threshold this is a very powerful solution to make the system robust against noise sources.
The exclusion of a sector of the hemispherical search grid is the preferred solution as it covers most noise sources without the need of defining each noise sources position. This is an easy way to hide noise sources with directional sound radiation while at the same time ensure detection of speaking persons. Furthermore it is possible to leave out specific areas where a disturbing noise source is located.
Another part of the invention solves the problem that appears if the exclusion of certain areas is not feasible e.g. if noise sources and speaking persons are located very close to each other. Many disturbing noise sources have most of their sound energy in certain frequency ranges, as depicted in
But even taken alone this technique is very powerful to reduce the chance of noise sources being detected by the source recognition algorithm. Dominant noise sources with a comparably narrow frequency band can be suppressed by excluding the appropriate frequency band from the SRP frequencies that are used for source detection. Broadband low Frequency noises can also be suppressed very well, as speech has a very wide frequency range and the source detection algorithms as presented works very robust even when only making use of higher frequencies.
Combining the above techniques allows for a manual or automated setup process, where noise sources are detected by the algorithm and either successively removed from the search grid, masked in the frequency range and/or hidden by locally applying a higher SRP-threshold.
SRP-PHAT detects a source for each frame of audio input data, independently from sources previously detected. This characteristic allows the detected source to suddenly change its position in space. This is a desired behavior if there are two sources reciprocally active shortly after each other and allows instant detection of each source. However, sudden changes of the source position might cause audible audio artifacts if the array is steered directly using the detected source positions, especially in situations where e.g. two sources are concurrently active. Furthermore it is not desirable to detect transient noise sources such as placing a coffee cup on a conference table or a coughing person. At the same time these noises cannot be tackled by the features described before.
The source detection unit makes use of different smoothing techniques in order to ensure an output that is free from audible artifacts caused by a rapidly steered beam and robust against transient noise sources while at the same time keeping the system fast enough to acquire speech signals without loss of intelligibility.
The signals captured by a multitude or array of microphones can be processed such that the output signal reflects predominant sound acquisition from a certain look direction while not being sensitive to sound sources of other directions not being the look direction. The resulting directivity response is called the beampattern the directivity around the look direction is called beam and the processing done in order to form the beam is the beamforming.
One way to process the microphone signals to achieve a beam is a Delay-and-sum beamformer. It sums all the microphone's signals after applying individual delays for the signal captured by each microphone.
A Delay-and-sum beamformer (DSB) has several drawbacks. Its directivity for low frequencies is limited by the maximum length of the array, as the array needs to be large in comparison to the wavelength in order to be effective. On the other hand the beam will be very narrow for high frequencies and thus introduces varying high frequency response if the beam is not precisely pointed to the source and possibly unwanted sound signature. Furthermore spatial aliasing will lead to sidelobes at higher frequencies depending on the microphone spacing. Thus the design of an array geometry is contrary, as good directivity for low frequencies requires a physically large array, while suppression of spatial aliasing requires the individual microphone capsules to be spaced as dense as possible.
In a filter-and-sum beamformer (FSB) the individual microphone signals are not just delayed and summed but, more generally, filtered with a transfer function and then summed. In the embodiment as shown in
By constraining the outer microphone signals to lower frequencies using shading filters, the effective array length of the array can be made frequency dependent as shown in
Both DSB and FIB are non-optimal beamformers. The “Minimum Variance Distortionless Response” (MVDR) technique tries to optimize the directivity by finding filters that optimize the SNR ratio of a source at a given position and a given noise source distribution with given constraints that limit noise. This enables better low frequency directivity but requires a computationally expensive iterative search for optimized filter parameters.
The microphone system comprises a multitude of techniques to further overcome the drawbacks of the prior art.
In a FIB as known from prior art, the shading filters need to be calculated depending on the look direction of the array. The reason is that the projected length of the array is changing with the sound incidence angle, as can be seen in
These shading filters however will be rather long and need to be computed or stored for each look direction of the array. The invention comprises a technique to use the advantages of a FIB while keeping the complexity very low by calculating fixed shading filters computed for the broadside configuration and factoring out the delays as known from a DSB, depending on the look direction. In this case the shading filters can be implemented with rather short finite impulse response (FIR) filters in contrast to rather long FIR filters in a typical FIB. Furthermore factoring out the delays gives the advantage that several beams can be calculated very easily as the shading filters need to be calculated once. Only the delays need to be adjusted for each beam depending on its look direction, which can be done without significant need for complexity or computational resources. The drawback is that the beam gets warped as shown in
In the embodiment of the invention as shown in
The microphone system according to the invention comprises another technique to further improve the performance of the created beam. Typically an array microphone either uses a DSB, FIB or MVDR beamformer. The invention combines the benefits of a FIB and MVDR solution by crossfading both. When crossfading between an MVDR solution, used for low frequencies and a FIB, used for high frequencies, the better low frequency directivity of the MVDR can be combined with the more consistent beam pattern at higher frequencies of the FIB. Using a Linkwitz-Riley crossover filter, as known e.g. from loudspeaker crossovers, maintains magnitude response. The crossfade can be implicitly done in the FIR coefficients without computing both beams individually and afterwards crossfading them. Thus only one set of filters has to be calculated.
Due to several reasons, the frequency response of a typical beam will, in practice, not be consistent over all possible look directions. This leads to undesired changes in the sound characteristics. To avoid this the invented microphone system comprises a steering dependent output equalizer 2460 that compensates for frequency response deviations of the steered beam as depicted in
According to an aspect of the invention, the knowledge of the resulting look direction 3 dB LD that results from using the initial look direction LD for calculating the delay values can be utilized for determining a “skewed look direction”: Instead of using the desired look direction as initial look direction LD for calculating the delay values, the skewed look direction is used for calculating the delay values, and the skewed look direction is chosen in a way that the resulting look direction 3 dB LD matches the desired look direction. The skewed look direction can be determined from the desired look direction in the direction recognition unit 2440 for instance by using a corresponding look-up table and possibly by a suitable interpolation.
According to a further aspect of the invention, the concept of the “skewed look direction” can also be applied to a linear microphone array where all microphone capsules are arranged along a straight line. This can be an arrangement of microphone capsules as shown in
The microphone system according to the invention allows for predominant sound acquisition of the desired audio source, e.g. a person talking, utilizing microphone array signal processing. In certain environments like very large rooms and thus very long distances of the source location to the microphone system or very reverberant situations, it might be desirable to have even better sound pickup. Therefore it is possible to combine more than one of the microphone systems in order to form a multitude of microphone arrays. Preferably each microphone is calculating a single beam and an automixer selects one or mixes several beams to form the output signal. An automixer is available in most conference system processing units and provides the simplest solution to combine multiple arrays. Other techniques to combine the signal of a multitude of microphone arrays are possible as well. For example the signal of several line and or planar arrays could be summed. Also different frequency bands could be taken from different arrays to form the output signal (volumetric beamforming).
While this invention has been described in conjunction with the specific embodiments outlined above, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, the preferred embodiments of the invention as set forth above are intended to be illustrative, not limiting. Various changes may be made without departing from the spirit and scope of the inventions as defined in the following claims.