To determine electrodes encoding acoustical details about the music, we fitted spectrotemporal receptive fields (STRFs) for all 2,379 artifact-free electrodes within the dataset, assessing how effectively the HFA recorded at these websites might be linearly predicted from the music’s auditory spectrogram (Fig 1). From a dense, bilateral, predominantly frontotemporal protection (Fig 2A), we recognized 347 electrodes with a big STRF (Fig 2B; see S1 Fig for an in depth view of every affected person’s protection and important electrodes). We discovered the next proportion of song-responsive electrodes in the fitting hemisphere. There have been 199 important electrodes out of 1,479 complete within the left hemisphere and 148 out of 900 in the fitting one (Fig 2B, 13.5% versus 16.4%, respectively; X² (1, N = 2,379) = 4.01, p = .045).

Fig 1. Protocol, information preparation, and encoding mannequin becoming.

(A) High: Waveform of the complete music stimulus. Contributors listened to a 190.72-second rock music (One other Brick within the Wall, Half 1, by Pink Floyd) utilizing headphones. Backside: Auditory spectrogram of the music. Orange bars on high symbolize elements of the music with vocals. (B) X-ray exhibiting electrode protection of 1 consultant affected person. Every dot is an electrode, and the sign from the 4 highlighted electrodes is proven in (C). (C) HFA elicited by the music stimulus in 4 consultant electrodes. (D) Zoom-in on 10 seconds (black bars in A and C) of the auditory spectrogram and the elicited neural exercise in a consultant electrode. Every time level of the HFA (y_i, crimson dot) is paired with a previous 750-ms window of the music spectrogram (X_i, black rectangle) ending right now level (proper fringe of the rectangle, in crimson). The set of all pairs (X_i, y_i), with i starting from .75 to 190.72 seconds represent the examples (or observations) used to coach and consider the linear encoding fashions. Linear encoding fashions used right here consist in predicting the neural exercise (y) from the auditory spectrogram (X), by discovering the optimum intercept (a) and coefficients (w). (E) STRF for the electrode proven in crimson in (B), (C), and (D). STRF coefficients are z-valued and are represented as w within the earlier equation. Be aware that 0 ms (timing of the noticed HFA) is on the proper finish of the x-axis, as we predict HFA from the previous auditory stimulus. The information underlying this determine might be obtained at https://doi.org/10.5281/zenodo.7876019. HFA, high-frequency exercise; STRF, spectrotemporal receptive discipline.

https://doi.org/10.1371/journal.pbio.3002176.g001

Fig 2. Anatomical location of song-responsive electrodes.

(A) Electrode protection throughout all 29 sufferers proven on the MNI template (N = 2,379). All offered electrodes are freed from any artifactual or epileptic exercise. The left hemisphere is plotted on the left. (B) Location of electrodes considerably encoding the music’s acoustics (N_sig = 347). Significance was decided by the STRF prediction accuracy bootstrapped over 250 resamples of the coaching, validation, and take a look at units. Marker coloration signifies the anatomical label as decided utilizing the FreeSurfer atlas, and marker dimension signifies the STRF’s prediction accuracy (Pearson’s r between precise and predicted HFA). We use the identical coloration code within the following panels and figures. (C) Variety of important electrodes per anatomical area. Darker hue signifies a right-hemisphere location. (D) Common STRF prediction accuracy per anatomical area. Electrodes beforehand labeled as supramarginal, different temporal (i.e., aside from STG), and different frontal (i.e., aside from SMC or IFG) are pooled collectively, labeled as different and represented in white/grey. Error bars point out SEM. The information underlying this determine might be obtained at https://doi.org/10.5281/zenodo.7876019. HFA, high-frequency exercise; IFG, inferior frontal gyrus; MNI, Montreal Neurological Institute; SEM, Normal Error of the Imply; SMC, sensorimotor cortex; STG, superior temporal gyrus; STRF, spectrotemporal receptive discipline.

https://doi.org/10.1371/journal.pbio.3002176.g002

Nearly all of the 347 important electrodes (87%) have been concentrated in 3 areas: 68% in bilateral superior temporal gyri (STG), 14.4% in bilateral sensorimotor cortices (SMCs, on the pre- and postcentral gyri), and 4.6% in bilateral IFG (Fig 2C). The proportion of song-responsive electrodes per area was 55.7% for STG (236 out of 424 electrodes), 11.6% for SMC (45/389), and seven.4% for IFG (17/229). The remaining 13% of great electrodes have been distributed within the supramarginal gyri and different frontal and temporal areas. To look at whether or not the upper proportion of song-responsive electrodes in the fitting hemisphere was pushed by completely different nonuniform coverages between each hemispheres (e.g., by a denser protection of nonauditory areas within the left hemisphere than in the fitting hemisphere), we restricted our evaluation to the three primary song-responsive areas (STG, SMC, and IFG). We discovered the next proportion of song-responsive electrodes in these proper song-responsive areas, with 133 important electrodes out of 374 complete, in opposition to 165 out of 654 within the corresponding left areas (35.6% versus 25.3%, respectively; X² (1, N = 1,026) = 12.34, p < .001).

Evaluation of STRF prediction accuracies (Pearson’s r) discovered a primary impact of laterality (additive two-way ANOVA with laterality and cortical areas as components; F(1, 346) = 7.48, p = 0.0065; Fig 2D), with increased correlation coefficients in the fitting hemisphere than within the left (M_R = .203, SD_R = .012; M_L = .17, SD_L = .01). We additionally discovered a primary impact of cortical areas (F(3, 346) = 25.09, p < .001), with the best prediction accuracies in STG (Tukey–Kramer submit hoc; M_STG = .266, SD_STG = .007; M_SMC = .194, SD_SMC = .017, p_STGvsSMC < .001; M_IFG = .154, SD_IFG = .027, p_STGvsSMC < .001; M_different = .131, SD_different = .016, p_STGvsSMC < .001). As well as, we discovered increased prediction accuracies in SMC in comparison with the group not together with STG and IFG (M_SMC = .194, SD_SMC = .017; M_different = .131, SD_different = .016, p_SMCvsOther = .035).

We examined music reconstruction from neural exercise and the way methodological components together with variety of electrodes included within the mannequin, the dataset length, or the mannequin sort used impacted decoding accuracy. We carried out a bootstrap evaluation by becoming linear decoding fashions on subsets of electrodes randomly sampled from all 347 important electrodes throughout the 29 sufferers, no matter anatomical location. This revealed a logarithmic relationship between what number of electrodes have been used as predictors within the decoding mannequin and the ensuing prediction accuracy (Fig 3A). For instance, 80% of one of the best prediction accuracy (utilizing all 347 important electrodes) was obtained with 43 (or 12.4%) electrodes. We noticed the identical relationship on the single-patient degree, for fashions skilled on every affected person’s important electrodes, though with decrease decoding accuracies (solid-colored circles in S2 Fig; for instance, 43 electrodes offered 66% of one of the best prediction accuracy). We noticed an identical logarithmic relationship between dataset length and prediction accuracy utilizing a bootstrap evaluation (Fig 3B). For instance, 90% of one of the best efficiency (utilizing the entire 190.72-second music) was obtained utilizing 69 seconds (or 36.1%) of information.

Fig 3. Music reconstruction and methodological issues.

(A) Prediction accuracy as a perform of the variety of electrodes included as predictors within the linear decoding mannequin. On the y-axis, 100% represents the utmost decoding accuracy, obtained utilizing all 347 important electrodes. The black curve reveals information factors obtained from a bootstrapping evaluation with 100 resamples for every variety of electrodes (with out alternative), whereas the crimson curve reveals a two-term energy collection match line. Error bars point out SEM. (B) Prediction accuracy as a perform of dataset length. (C) Auditory spectrograms of the unique music (high) and of the reconstructed music utilizing both linear (center) or nonlinear fashions (backside) decoding from all responsive electrodes. This 15-second music excerpt was held out throughout hyperparameter tuning by means of cross-validation and mannequin becoming and used solely as a take a look at set to judge mannequin efficiency. Corresponding audio waveforms have been obtained by means of an iterative phase-estimation algorithm and might be listened to in S1, S2, and S3 Audio information, respectively. Common efficient r-squared throughout all 128 frequency bins is proven above each decoded spectrograms. (D) Auditory spectrogram of the reconstructed music utilizing nonlinear fashions from electrodes of affected person P29 solely. Corresponding audio waveform might be listened to in S4 Audio. The information underlying this determine might be obtained at https://doi.org/10.5281/zenodo.7876019. SEM, Normal Error of the Imply.

https://doi.org/10.1371/journal.pbio.3002176.g003

Relating to mannequin sort, linear decoding offered a mean decoding accuracy of .325 (median of the 128 fashions’ efficient r-squared; IQR .232), whereas nonlinear decoding utilizing a two-layer, absolutely related neural community (multilayer perceptron (MLP)) yielded a mean decoding accuracy of .429 (IQR .222). This 32% improve in efficient r-squared (+.104 from .325) was important (two-sided paired t take a look at, t(127) = 17.48, p < .001). Consistent with this increased efficient r-squared for MLPs, the decoded spectrograms revealed variations between mannequin varieties, with the nonlinear reconstruction (Fig 3C, backside) exhibiting finer spectrotemporal particulars, relative to the linear reconstruction (Fig 3C, center). General, the linear reconstruction (S2 Audio) sounded muffled with sturdy rhythmic cues on the presence of foreground parts (vocals syllables and lead guitar notes); a way of spectral construction underlying timbre and pitch of lead guitar and vocals; a way of concord (chord development transferring from Dm to F, C, and Dm); however restricted sense of the rhythm guitar sample. The nonlinear reconstruction (S3 Audio) offered a recognizable music, with richer particulars as in comparison with the linear reconstruction. Perceptual high quality of spectral parts reminiscent of pitch and timbre have been particularly improved, and phoneme id was perceptible. There was additionally a stronger sense of concord and an emergence of the rhythm guitar sample.

Stimulus reconstruction was additionally relevant to a single affected person with high-density 3-mm electrode spacing protection. We used nonlinear fashions to reconstruct the music from the 61 important electrodes of affected person P29 (Fig 3D). These fashions carried out higher than the linear reconstruction based mostly on electrodes from all sufferers (efficient r-squared of .363), however decoding accuracy was decrease than that obtained with 347 important electrodes from all sufferers. On the perceptual aspect, these single-patient–based mostly fashions offered a degree of spectrotemporal particulars excessive sufficient to acknowledge the music (S4 Audio). To evaluate the decrease certain of single-patient–based mostly decoding, we reconstructed the music from the neural exercise of three extra sufferers (P28, P15, and P16), with fewer electrodes (23, 17, and 10, respectively, versus 61 in P29) and a decrease density (1 cm, 6 mm, and 1 cm center-to-center electrode distance, respectively, versus 3 mm in P29), however nonetheless protecting song-responsive areas (largely proper, left, and left STG, respectively) and with an excellent linear decoding accuracy (Pearson’s r = .387, .322, and .305, respectively, versus .45 in P29). Nonlinear fashions reconstructed the music spectrogram with an efficient r-squared of .207, .257, and .166, respectively (S3 Fig). Within the reconstructed waveforms (S5, S6, and S7 Audio information), we retrieved partial vocals (e.g., in P15, “all,” “was,” and “only a brick” have been the one recognizable syllables, as might be seen within the reconstructed spectrogram; S3 Fig, high) and a way of concord, though with various focus in recognizability.

We then quantified the decoded music recognizability by correlating excerpts of the unique versus decoded music spectrograms. Each linear (Fig 4A) and nonlinear (Fig 4B) reconstructions offered a excessive proportion of right identifications (32/38 and 36/38, respectively; Fig 4, left panels) and important identification imply percentiles (95.2% and 96.3%, respectively; Fig 4, proper panels; 1,000-iteration permutation take a look at, CI95 [.449 .582] for linear, [.447 .583] for nonlinear).

Fig 4. Music-excerpt identification rank evaluation.

After decoding the entire music by means of 12 distinct 15-second take a look at units, we divided each the unique music and the decoded spectrogram into 5-second excerpts and computed the correlation coefficient for all doable original-decoded pairs. (A) Decoding utilizing linear fashions. Left panel reveals the correlation matrix, with crimson dots indicating the row-wise most values (e.g., first decoded 5-second excerpt correlates most with thirty second unique music excerpt). Proper panel reveals a histogram of the excerpt identification rank, a measure of how shut the utmost original-decoded correlation coefficient landed from true excerpt determine (e.g., third original-decoded pair correlation coefficient, on the matrix diagonal, was the second highest worth on the third excerpt’s row, thus ranked 37/38). Grey shaded space represents the 95% confidence interval of the null distribution estimated by means of 1,000 random permutations of the unique music excerpt identities. The crimson vertical line reveals the typical identification rank throughout all music excerpts. (B) Similar panels for decoding utilizing nonlinear fashions. The information underlying this determine might be obtained at https://doi.org/10.5281/zenodo.7876019.

https://doi.org/10.1371/journal.pbio.3002176.g004

We analyzed STRF coefficients for all 347 important electrodes to evaluate how completely different musical parts have been encoded in numerous mind areas. This evaluation revealed quite a lot of spectrotemporal tuning patterns (Fig 5A; see S4 Fig for an in depth view of affected person P29’s STRFs computed for his or her 10-by-25, 3-mm-center-to-center grid of electrodes). To totally characterize the connection between the music spectrogram and the neural exercise, we carried out an unbiased part evaluation (ICA) on all important STRFs. We recognized 3 elements with distinct spectrotemporal tuning patterns, every explaining greater than 5% variance and collectively explaining 52.5% variance (Fig 5B).

Fig 5. Evaluation of the STRF tuning patterns.

(A) Consultant set of 10 STRFs (out of the 347 important ones) with their respective places on the MNI template utilizing matching markers. Coloration code is similar to the one utilized in Fig 1. (B) Three ICA elements every explaining greater than 5% variance of all 347 important STRFs. These 3 elements present onset, sustained, and late onset exercise. Percentages point out defined variance. (C) ICA coefficients of those 3 elements, plotted on the MNI template. Coloration code signifies coefficient amplitude, with in crimson the electrodes which STRFs symbolize the elements essentially the most. (D) To seize tuning to the rhythm guitar sample (Sixteenth notes at 100 bpm, i.e., 6.66 Hz), pervasive all through the music, we computed temporal modulation spectra of all important STRFs. Instance modulation spectrum is proven for a proper STG electrode. For every electrode, we extracted the utmost temporal modulation worth throughout all spectral frequencies round a price of 6.66 Hz (crimson rectangle). (E) All extracted values are represented on the MNI template. Electrodes in crimson present tuning to the rhythm guitar sample. The information underlying this determine might be obtained at https://doi.org/10.5281/zenodo.7876019. ICA, unbiased part evaluation; MNI, Montreal Neurological Institute; STG, superior temporal gyrus; STRF, spectro-temporal receptive discipline.

https://doi.org/10.1371/journal.pbio.3002176.g005

The primary part (28% defined variance) confirmed a cluster of optimistic coefficients (in crimson, in Fig 5B, high row) spreading over a broad frequency vary from about 500 Hz to 7 kHz and over a slim time window centered round 90 ms earlier than the noticed HFA (positioned at time lag = 0 ms, on the proper fringe of all STRFs). This temporally transient cluster revealed tuning to sound onsets. This part, known as the “onset part,” was discovered solely in electrodes positioned in bilateral posterior STG (Fig 5C, high row, electrodes depicted in crimson). Fig 6C, high row, confirmed in crimson the elements of the music eliciting the best HFA improve in electrodes possessing this onset part. These elements corresponded to onsets of lead guitar or synthesizer motifs (Fig 6A, blue and purple bars, respectively; see Fig 6E for a zoom-in) performed each 2 bars (inexperienced bars) and to onsets of syllable nuclei within the vocals (orange bars; see Fig 6D for a zoom-in).

Fig 6. Encoding of musical parts.

(A) Auditory spectrogram of the entire music. Orange bars above the spectrogram mark all elements with vocals. Blue bars mark lead guitar motifs, and purple bars mark synthesizer motifs. Inexperienced vertical bars delineate a collection of eight 4/4 bars (or measures). Thicker orange and blue bars mark places of the zoom-ins offered in (D) and (E), respectively. (B) Three STRF elements as offered in Fig 5B, specifically onset (high), sustained (center), and late onset (backside). (C) Output of the sliding correlation between the music spectrogram (A) and every of the three STRF elements (B). Optimistic Pearson’s r values are plotted in crimson, marking elements of the music that elicited a rise of HFA in electrodes exhibiting the given part. Be aware that for the sustained plot (center), optimistic correlation coefficients are particularly noticed throughout vocals. Additionally, observe for each the onset and late onset plots (high and backside, respectively), optimistic r values within the second half of the music correspond to steer guitar and synthesizer motifs, occurring each different 4/4 bar. (D) Zoom-in on the third vocals. Lyrics are offered above the spectrogram, decomposed into syllables. Most syllables triggered an HFA improve in each onset and late onset plots (high and backside, respectively), whereas a sustained improve of HFA was noticed throughout the complete vocals (center). (E) Zoom-in on a lead guitar motif. Sheet music is offered above the spectrogram. Most notes triggered an HFA improve in each onset and late onset plots (high and backside, respectively), whereas there was no HFA improve for the sustained part (center). The information underlying this determine might be obtained at https://doi.org/10.5281/zenodo.7876019. HFA, high-frequency exercise; STRF, spectrotemporal receptive discipline.

https://doi.org/10.1371/journal.pbio.3002176.g006

The second part (14.7% defined variance) confirmed a cluster of optimistic coefficients (in crimson, in Fig 5B, center row) spreading over the complete 750-ms time window and over a slim frequency vary from about 4.8 to 7 kHz. This part, known as the “sustained part,” was present in electrodes positioned in bilateral mid- and anterior STG and in bilateral SMC (Fig 5C, center row). Additionally, this part correlated greatest with elements of the music containing vocals, thus suggesting tuning to speech (Fig 6C, center row, in crimson; see Fig 6D for a zoom-in).

The third part (9.8% defined variance) confirmed an identical tuning sample because the onset part, solely with an extended latency of about 210 ms earlier than the noticed HFA (Fig 5B, backside row). This part, referred to any further because the “late onset part,” was present in bilateral posterior and anterior STG, neighboring the electrodes representing the onset part, and in bilateral SMC (Fig 5C, backside row). As with the onset part, this late onset part was most correlated with onsets of lead guitar and synthesizer motifs and of syllable nuclei within the vocals, solely with an extended latency (Fig 6C, backside row; see Fig 6D and 6E for zoom-ins).

A fourth part was discovered by computing the temporal modulations and extracting the utmost coefficient round a price of 6.66 Hz for all 347 STRFs (Fig 5D, crimson rectangle). This price corresponded to the Sixteenth notes of the rhythm guitar, pervasive all through the music, on the music tempo of 99 bpm (beats per minute). It was translated within the STRFs as small clusters of optimistic coefficients spaced by 150 ms (1/6.66 Hz) from one another (e.g., Fig 5A, electrode 5). This part, known as the “rhythmic part,” was present in electrodes positioned in bilateral mid-STG (Fig 5E).

To evaluate the function of those completely different cortical areas and practical elements in representing musical options, we carried out an ablation evaluation utilizing linear decoding fashions. We first computed linear decoding fashions for every of the 32 frequency bins of the music spectrogram, utilizing the HFA of all 347 important electrodes as predictors. This yielded a mean prediction accuracy of .62 (Pearson’s r; min .27—max .81). We then eliminated (or ablated) anatomically or functionally outlined units of electrodes and computed a brand new collection of decoding fashions to evaluate how every ablation would affect the decoding accuracy. We used prediction accuracies of the total, 347-electrode fashions as baseline values (Fig 7). We discovered a big primary impact of electrode units (one-way ANOVA; F [1, 24] = 78.4, p < .001). We then ran a collection of submit hoc analyses to look at the affect of every set on prediction accuracy.

Fig 7. Ablation evaluation on linear decoding fashions.

We carried out “digital lesions” within the predictors of decoding fashions, by ablating both anatomical (A) or practical (B) units of electrodes. Ablated units are proven on the x-axis, and their impacts on the prediction accuracy (Pearson’s r) of linear decoding fashions, as in comparison with the efficiency of a baseline decoding mannequin utilizing all 347 important electrodes, are proven on the y-axis. For every ablation, a notched field plot represents the distribution of the modifications in decoding accuracy for all 32 decoding fashions (one mannequin per frequency bin of the auditory spectrogram). For every field, the central mark signifies the median; the notch delineates the 95% confidence interval of the median; backside and high field edges point out the twenty fifth and seventy fifth percentiles, respectively; whiskers delineate the vary of nonoutlier values; and circles point out outliers. Crimson asterisks point out important affect from ablating a given set of electrodes. The information underlying this determine might be obtained at https://doi.org/10.5281/zenodo.7876019.

https://doi.org/10.1371/journal.pbio.3002176.g007

Sufferers passively listened to the music One other Brick within the Wall, Half 1, by Pink Floyd (launched on the album The Wall, Harvest Data/Columbia Data, 1979). They have been instructed to pay attention attentively to the music, with out specializing in any particular element. Whole music length was 190.72 seconds (waveform is represented in Fig 1A, high). The auditory stimulus was digitized at 44.1 kHz and delivered by means of in-ear monitor headphones (bandwidth 12 Hz to 23.5 kHz, 20 dB isolation from surrounding noise) at a cushty sound degree adjusted for every affected person (50 to 60 dB SL). Eight sufferers had multiple recording of the current job, wherein instances we chosen the cleanest one (i.e., containing the least epileptic exercise or noisy electrodes).

Direct cortical recordings have been obtained by means of grids or strips of platinum-iridium electrodes (Advert-Tech Medical, Oak Creek, WI), with center-to-center distances of 10 mm for 21 sufferers, 6 mm for 4, 4 mm for 3, or 3 mm for one. We recruited sufferers within the research if their implantation map coated at the least partially the STG (left or proper). The cohort consists of 28 unilateral instances (18 left, 10 proper) and 1 bilateral case. Whole variety of electrodes throughout all 29 sufferers was 2,668 (vary 36 to 250, imply 92 electrodes). ECoG exercise was recorded at a sampling price of 1,200 Hz utilizing g.USBamp biosignal acquisition units (g.tec, Graz, Austria) and BCI2000 [68].

To review the connection between the acoustics of the auditory stimulus and the ECoG-recorded neural exercise, the music waveform was remodeled right into a magnitude-only auditory spectrogram utilizing the NSL MATLAB Toolbox [69]. This transformation mimics the processing steps of early levels of the auditory pathways, from the cochlea’s spectral filter financial institution to the midbrain’s decreased higher restrict of phase-locking skill, and outputs a psychoacoustic-, neurophysiologic-based spectrotemporal illustration of the music (just like the cochleagram) [70,71]. The ensuing auditory spectrogram has 128 frequency bins from 180 to 7,246 Hz, with attribute frequencies uniformly distributed alongside a logarithmic frequency axis (24 channels per octave), and a sampling price of 100 Hz. This full-resolution, 128-frequency bin spectrogram is used within the music reconstruction evaluation. For all different analyses, to lower the computational load and the variety of options, we output a decreased spectrogram with 32 frequency bins from 188 to six,745 Hz (Fig 1A, backside).

We used the HFA (70 to 150 Hz) as an estimate of native neural exercise [72] (Fig 1C). For every dataset, we visually inspected uncooked recorded indicators and eliminated electrodes exhibiting noisy or epileptic exercise, with the assistance of a neurologist (RTK). General, from our beginning set of two,668 electrodes, we eliminated 106 noisy electrodes (absolute vary 0 to 22, imply 3.7 electrodes; relative vary 0% to twenty.2%, imply 3.7%) and 183 epileptic electrodes (absolute vary 0 to twenty-eight, imply 6.3; relative vary 0% to 27.6%, imply 7.6%) and obtained a set of two,379 artifact-free electrodes.

We then extracted information aligned with the music stimulus, including 10 seconds of information padding earlier than and after the music (to stop filtering-induced edge artifacts). We filtered out power-line noise, utilizing a variety of notch filters centered at 60 Hz and harmonics as much as 300 Hz (Butterworth, fourth order, 2 Hz bandwidth), and eliminated sluggish drifts with a 1-Hz high-pass filter (Butterworth, fourth order). We used a bandpass–Hilbert method [73] to extract HFA, with 20-Hz-wide sub-bands spanning from 70 to 150 Hz in 5-Hz steps (70 to 90, 75 to 95, … as much as 130 to 150 Hz). We selected a 20-Hz bandwidth to allow the commentary of temporal modulations as much as 10 Hz [74], encompassing the 6.66-Hz Sixteenth-note rhythm guitar sample, pervasive all through the music. This constitutes a vital methodological level, enabling the commentary of the rhythmic part (Fig 3D). For every sub-band, we first bandpass-filtered the sign (Butterworth, fourth order), then carried out median-based frequent common reference (CAR) [75], and computed the Hilbert rework to acquire the envelope. We standardized every sub-band envelope utilizing sturdy scaling on the entire time interval (subtracting the median and dividing by the interquartile vary between the tenth and ninetieth percentiles) and averaged them collectively to yield the HFA estimate. We carried out CAR individually for electrodes plugged on completely different splitter containers to optimize denoising in 14 contributors. Lastly, we eliminated the 10-second pads, down-sampled information to 100 Hz to match the stimulus spectrogram’s sampling price, and tagged outlier time samples exceeding 7 customary deviations for later removing within the modeling preprocessing. We used FieldTrip [76] (model from Could 11, 2021) and selfmade scripts to carry out all of the above preprocessing steps. Until specified in any other case, all additional analyses and computations have been applied in MATLAB (The MathWorks, Natick, MA, USA; model 2021a).

We adopted the anatomical information processing pipeline offered in Stolk and colleagues [77] to localize electrodes from a preimplantation MRI, a postimplantation CT scan, and protection info mapping electrodes to channel numbers within the practical information. After coregistration of the CT scan to the MRI, we carried out brain-shift compensation with a hull obtained utilizing scripts from the iso2mesh toolbox [78,79]. Cortical surfaces have been extracted utilizing the FreeSurfer toolbox [80]. We used volume-based normalization to transform patient-space electrode coordinates into MNI coordinates for illustration functions, and surface-based normalization utilizing the FreeSurfer’s fsaverage template to robotically acquire anatomical labels from the aparc+aseg atlas. Labels have been then confirmed by a neurologist (RTK).

We then computed the characteristic lag matrix from the music’s auditory spectrogram (Fig 1D). As HFA is elicited by the music stimulus, we intention at predicting HFA from the previous music spectrogram. We selected a time window between 750 ms and 0 ms earlier than HFA to permit a ample temporal integration of auditory-related neural responses, whereas guaranteeing an inexpensive features-to-observations ratio to keep away from overfitting. This resulted in 2,400 options (32 frequency bins by 75 time lags at a sampling price of 100 Hz).

Then, we mixed these 250 split-scale-fit-evaluate iterations in a bootstrap-like method to acquire 1 STRF and assess its significance (i.e., whether or not we are able to linearly predict HFA, at a given electrode, from the music spectrogram). For every STRF, we z-scored every coefficient throughout the 250 fashions (Fig 1E). For the prediction accuracy, we computed the 95% confidence interval (CI) from the 250 correlation coefficients and deemed an electrode as important if its 95% CI didn’t comprise 0. As a further criterion, we rejected important electrodes with a mean r-squared (throughout the 250 fashions) at or under 0.

To research the affect of mannequin sort on decoding accuracy and to evaluate the extent to which we might reconstruct a recognizable music, we skilled linear and nonlinear fashions to decode every of the 128 frequency bins of the total spectral decision music spectrogram from HFA of all 347 important electrodes. We used the MLP—a easy, absolutely related neural community—as a nonlinear mannequin (MLPRegressor in sklearn). We selected an MLP structure of two hidden layers of 64 items every, based mostly each on an extension of the Common Approximation Theorem stating {that a} 2 hidden layer MLP can approximate any steady multivariate perform [38] and on a earlier research with an identical use case [35]. Since MLP layers are absolutely related (i.e., every unit of a layer is related to all items of the following layer), the variety of coefficients to be fitted is drastically elevated comparatively to linear fashions (on this case, F * N + N * N + N vs. F, respectively, the place the overall variety of options F = E * L, with E representing the variety of important electrodes included as inputs of the decoding mannequin, and L the variety of time lags, and N represents the variety of items per layer). Given the restricted dataset length, we decreased time lags to 500 ms based mostly on the absence of great exercise past this level within the STRF elements and used this L worth in each linear and nonlinear fashions.

To discover the extent to which we might reconstruct the music from neural exercise, we collected the 128 predicted targets for each linear and MLP decoding fashions as computed above, subsequently assembling the decoded auditory spectrograms. To denoise and enhance sound high quality, we raised all spectrogram samples to the facility of two, thus highlighting distinguished musical parts reminiscent of vocals or lead guitar chords, relative to background noise. As each magnitude and section info are required to reconstruct a waveform from a spectrogram, we used an iterative phase-estimation algorithm to rework the magnitude-only decoded auditory spectrogram into the music waveform (aud2wav) [69]. To have a good foundation in opposition to which we might examine the music reconstruction of the linearly and nonlinearly decoded spectrograms, we remodeled the unique music excerpt akin to the mounted take a look at set into an auditory spectrogram, discarded the section info, and utilized this algorithm to revert the spectrogram right into a waveform (S1 Audio). We carried out 500 iterations of this aud2wav algorithm, sufficient to achieve a plateau the place error didn’t enhance additional.

To judge the standard of the decoded music spectrogram, we used an goal method based mostly on correlation [34]. Firstly, we divided the music into twelve 15-second segments. We then decoded every one among these segments as held-out take a look at units, utilizing each linear and nonlinear fashions. Subsequent, we divided all predicted 15-second spectrograms into 5-second excerpts. We computed the 2D correlation coefficients between every of the 38 decoded excerpts and all 38 unique excerpts. We then carried out an excerpt identification rank evaluation by sorting these coefficients in ascending order and by figuring out the rank of the particular excerpt correlation. For instance, for decoded excerpt #4, if the correlation coefficient with the unique excerpt #4 is the third greatest, its rank will likely be 36/38. The ensuing metric ranges from 1/38 (worst doable identification, i.e., the given decoded excerpt matches greatest with all different music excerpts than with its corresponding one) to 38/38, and averaging ranks throughout all music excerpts provides a proxy for classification skill of the linear and nonlinear fashions. To evaluate statistical significance, we computed 1,000 iterations of the algorithm above whereas randomly permuting indices of the unique music excerpts to acquire a null distribution of the imply normalized rank. We deemed the imply normalized rank of our linear and nonlinear decoding fashions as important in the event that they have been outdoors of the 95% CI (i.e., exceeded the 97.5 percentile) of the null distribution.

To evaluate the contribution of various mind areas and STRF elements in representing the music, we carried out an ablation evaluation. We quantified the affect of ablating units of electrodes on the prediction accuracy of a linear decoding mannequin computed utilizing all 347 important electrodes. Firstly, we constituted units of electrodes based mostly on anatomical or practical standards. We outlined 12 anatomical units by combining 2 components—space (entire hemisphere, STG, SMC, IFG, or different areas) and laterality (bilateral, left or proper). We outlined 12 practical units by combining 2 components—STRF part recognized within the STRF coefficient analyses (onset, sustained, late onset, and rhythmic) and laterality (bilateral, left or proper). See Fig 5 for the precise record of electrode units. Secondly, we computed the decoding fashions utilizing the identical algorithm as for the encoding fashions. Decoding fashions intention at predicting the music spectrogram from the elicited neural exercise. Right here, we used HFA from a set of electrodes as enter and a given frequency bin of the music spectrogram as output. For every of the 24 ablated units of electrodes, we obtained 32 fashions (1 per spectrogram frequency bin) and in contrast every one among them to the corresponding baseline mannequin computed utilizing all 347 important electrodes (repeated-measure one-way ANOVA). We then carried out a a number of comparability (submit hoc) take a look at to evaluate variations between ablations.

Be aware that we carried out this ablation evaluation on linear decoding fashions to make sure interpretability of the outcomes. Certainly, as deep neural networks are in a position to mannequin any perform [38], they might reconstitute acoustic info (e.g., when ablating STG) from higher-order, nonlinear illustration of musical info (e.g., in SMC or IFG) and will thus alleviate, if not masks completely, any affect on decoding accuracy. Utilizing linear decoding fashions restricts compensation to the identical (or at most, linearly associated) info degree and allows drawing conclusions from the ablation evaluation outcomes. Additional, in comparison with linear fashions, nonlinear fashions require tuning extra hyperparameters, with most probably completely different optimum values between ablations, which might bias the outcomes.