Multichannel parallel processing of neural signals in memristor arrays | Science Advances

A fully implantable neural interface with a large number of recording channels brings the gospel to patients with motor or speech loss. With the rapid increase in the number of recording channels, conventional complementary metal oxide semiconductor (CMOS) chips used for neural signal processing face severe challenges in terms of parallelism, scalability, computational cost, and power consumption. In this work, we propose a previously unexplored method to utilize the rich dynamic characteristics of the memristor array to process multi-channel neural signals in the memristor array in parallel. Extract the key information of the neural signal waveform and encode it in the memristor conductance modulation. Develop signal splitting scheme to adapt to equipment changes. In order to verify the fidelity of the processing results, and further prove the epileptic seizure prediction, compared with CMOS similar products, the epilepsy prediction has a high accuracy of more than 95%, and the power efficiency is increased by more than 1000 times. This work shows that the memristor array may be a promising multi-channel signal processing module for implantable neural interfaces in the future.

Electrical monitoring of brain activity has been proven to be an effective method to explore underlying neural principles, and has a wide range of applications in the field of biomedicine (

-

). Invasive neural probes, microelectrode arrays or electrode grids with huge recording channels are popular tools for recording brain activity (

). In recent years, with the exponential increase in the number of recording channels, the amount of recorded neural signal data has exploded (

,

), which poses a major challenge for real-time processing of multi-channel neural signals (

). Most existing signal processing hardware systems rely on multiplexing multiple recording channels into a single or several processing units to perform heavy computing tasks (

) (

). However, these systems have inherent scalability issues due to the delay and power consumption caused by multiplexing. These problems further limit many practical biomedical applications that usually require more recording channels, such as disease diagnosis and neural interfaces. Under these circumstances, new equipment and hardware systems are urgently needed to process multi-channel neural signals in parallel, especially those that can directly extract and store key information from neural signals in an efficient manner.

(

) An illustration of a conventional neural signal processing system that usually uses a multiplexer (MUX) to convert multi-channel neural signals into serial signals and then process them for biomarker calculations. (

) The proposed memristor-based system, which uses a memristor array to process multi-channel neural signals in parallel, in which biomarker extraction is achieved by the memristor conductance modulation. BL, bit line; WL, word line; SL, source code line.

Memristor (Memristor) is an emerging neuromorphic device, due to its ability to use inherent physical properties to perform calculations (

). In the literature, vector matrix multiplication based on the memristor array based on Ohm's law and Kirchhoff's law has been widely proven (

). In addition, the rich dynamic characteristics of memristors related to physical state changes can be used to implement logical operations (

) And time correlation detection (

), the calculation result is usually directly stored in the device. The latter method attracts low-power real-time neural signal processing. The early suggestion of this application is to use TaO's inherent super-threshold integration feature to detect neuronal peaks

Memristor (

). However, due to the inevitable differences between devices and the saturation behavior of device resistance (

), it is difficult to directly apply this method to multiple recording channels for parallel processing. In addition, processing another important neural signal-local potential (LFP) (

), which reflects the sum of postsynaptic potentials (

), for neuroscience and epilepsy diagnosis (

). Due to the relatively low sampling rate of LFP, it may be more suitable for low-power biomedical electronics than neuron spikes (

). It also has richer signal dynamics than neuronal spikes. Therefore, more complex calculations are needed to extract the key information, instead of amplitude threshold detection like neuron spikes (

). Intracranial electroencephalogram (iEEG) is a typical method of recording LFP by using electrodes on or inside the brain, and iEEG signals are usually collected from epilepsy patients (

), which is one of the most common neurological diseases (

). The prediction of seizures helps to develop new epilepsy treatment strategies, and it is necessary to be able to distinguish between pre-seizure and inter-seizure states in order to warn patients before seizures or provide early treatment (

). For this reason, the multi-channel parallel processing of LFP based on memristor dynamics may be an attractive method for predicting epileptic seizures. So far, it has not been experimentally proven.

Here, we propose a system that uses one transistor-one resistor (1T1R) memristor array for parallel processing of multi-channel LFP (

And figure. S1). The system uses inherent memristor conductance modulation to extract the energy and change information of the input nerve signal. The signal segmentation scheme in the memristor array is implemented through the 1T1R architecture to retain more input information and can also adapt to differences between devices. In order to verify the feasibility of the proposed system, a 16-channel iEEG signal was used as the neural signal to be processed. In addition, high-precision epileptic seizure prediction based on the processing results is demonstrated.

1k cell crosspoint array of memristor with TiN/TaO

/ HfO

/ TiN material stack is manufactured (for manufacturing details, please refer to materials and methods) as a platform for parallel processing of multi-channel neural signals. Here, TaO

The layer is used as a thermal enhancement layer to improve the analog conductance modulation behavior of HfO

Based on the memristor, its conductance state can be gradually adjusted by applying electrical stimulation. Each unit has a 1T1R structure, as shown in Figure 2. S2A, the memristor stack is located on top of the transistor drain terminal [bit line (BL)]. The gate and source terminals are connected to the word (WL) and source line (SL), respectively.

Shows the analog conductance modulation by SL under 15 identical RESET voltage pulses (

= 1.3 to 1.8 V,

= 5.0 V, and

= 0 V) ​​Start from the same initial conductance value. Similarly, figure. S2B displays the result of the SET process. Different colored lines represent different pulse amplitudes. It can be seen that the conductance modulation process of the memristor is highly related to the amplitude of the input pulse. When the pulse amplitude increases, the conductance decreases faster and tends to saturate at lower values, so the memristor can distinguish different input signal energies.

) The memristor conductance modulation process under the same RESET pulse starting from the same initial conductance value. Each data point is taken from the average of 128 devices.

= 5.0 V and

= 0V. The pulse width is 50 ns. (

) The waveforms of several designed input signal pulse trains. They have the same average amplitude of 1.5 V, but the amplitude of change (σ) increases from top to bottom. σ unit: volt. (

) The pulse train applied in (B) causes the evolution of the memristor conductance. Each data point is taken from the average of 128 devices. (

) The average change in conductance (Δ

, Left

Axis) After applying different pulse sequences, Δ is displayed

Increase with the change of input signal (right

axis).

In addition to signal energy, memristors can also distinguish between different input signal waveforms, such as changes. In order to verify this statement, four sets of 15 pulse sequences are used, which have the same 1.5 V average amplitude (ie, the same signal energy), but have different SDσ (ie, different signal changes) to modulate the memristor conductance (

). As we have seen, for different input signal waveforms, different electrical directing processes have been observed (

). Relatively large conductance change (Δ

) Is usually observed after the pulse with higher amplitude in each pulse sequence (indicated by the arrow), so a signal with a larger change will result in a larger Δ

In the final conductance, as shown in the figure

. This conductance modulation behavior is essential for neural signal processing because it helps to extract key information about changes in energy and input signals and encode them as changes in memristor conductance. It is worth noting that this important information is automatically stored in the non-volatile memristor and can be accessed as the conductance state of the device through a read operation for later feature extraction. This is the basis for subsequent parallel processing of neural signals.

In order to process multi-channel neural signals in parallel, a signal segmentation scheme was developed based on the memristor array, as shown in the figure.

. The original signal collected from the brain is first linearly converted (with amplification and offset) to the desired voltage range (ie ~1 to 2 V), and then used as the input signal of the memristor array, and then

Illustration of a signal segmentation scheme in a memristor array.

) Typical 16-channel waveforms of seizure and seizure signal clips. Blue, the middle. Red, paroxysmal. The sampling rate is 400 Hz. (

with

) Input voltage histogram after linear transformation of the original interictal and pre-ictal neural signals. Gain = 10

= 1.5V. (

) The conductance changes after processing the inter-wall and inter-wall signal segments in the memristor array respectively. (

) Histogram of conductance change (Δ

) Are located in (E) and (F) respectively.

The converted multi-channel signals (each lasting about 2.4 s) are fed into multiple rows of the memristor array. Then the WL turns on these 64 columns in turn to divide the input signal in each channel into segments with the same length (37.5 ms) to be applied to the corresponding memristor column, whose conductance changes record energy and change information. input signal. Here, the signal sampling rate is 400 Hz, so each segment corresponds to a 15-pulse sequence, similar to

. This signal segmentation scheme in the memristor array with 1T1R architecture allows it to store information about the evolution of the input signal over time. In fact, the size of the required memristor array can be determined by the number of recording channels, the length and the sampling rate of the neural signal to be processed and the working conditions of the memristor.

An example of a set of 16-channel beat and beat iEEG signals from the Kaggle epileptic seizure prediction dataset (for details on the dataset, see Materials and Methods) is shown in

(B and C). In order to obtain more recording channels and longer signal duration in practical applications, the size of the memristor array can be increased proportionally. Distribution of input voltage (linear conversion from original data, same gain, gain = 10)

And offset

= 1.5 V), compare the signals before and after the two groups

.

Plot Δ

The two signals are processed in parallel in the memristor array and then mapped. Distribution of output Δ

show on

. It is found that the average value of the inserted signal (~1.5 V) is similar, but the SD signal (0.24 V vs. 0.15 V) is much larger, resulting in a larger change in the conductivity of the memristor device (for example |Δ)

|> 15μS).

In order to further optimize the neural signal processing system based on the memristor array, a systematic study was carried out using different experimental conditions. For example, figure. When using input signals for SET and RESET operations, S3 compares system performance. It was found that the RESET operation resulted in a much larger contrast between intra-signal and inter-signal correlations, partly because the RESET process was more gradual than the SET process.

And figure. S1B. We should mention that the system performance may also be affected by the characteristics of the memristor (such as the on/off ratio, the number of states, and the linearity of the conductance modulation; see the discussion in Figure S3) and linear transformation conditions (such as the deviation of the neural signal). And amplification gain). For example, figure. S4A shows the correlation as a function of amplification gain.

(

= 1.5 V). Within 0.7×10

To 1.7×10

, The greater the gain, the greater the correlation contrast, and therefore the better the performance. Similarly, figure. S4B shows the correlation with

(Earnings = 10

). In the range of 1.2 to 1.5 V, the smaller the offset, the better the performance. All these results show that there is a lot of room for optimization to improve the system performance in practical applications. Further research is needed to further study the influence of memristor conductance modulation characteristics and neural signal linear transformation conditions on system performance.

For practical applications, the robustness of the multi-channel neural signal processing system based on the memristor array will be further analyzed on the time and space scale. The robustness of the time scale has been verified by conducting 200 trials, in which the first 100 wakes use the same seizure signal, and the other 100 wakes use the same seizure signal. All signals are processed in the same memristor array.

Display the correlation diagram between the corresponding Δ

Figure (For calculation of correlation coefficient, please refer to Materials and Methods). The inter-signal correlation is very low (<0.2), while the intra-signal correlation is very high (> 0.7). This result confirms that the processed neural signal in the memristor array retains sufficient information about the resulting Δ

Can be used to distinguish between seizures and inter-seizure signals. In practical applications, it is also important that the memristor maintain its characteristics during repeated operations to process a large amount of neural signal data from the brain. For this reason, we investigated the durability characteristics of the memristor, and the results showed that it can reach more than 2×10

Cycle (Figure S5). Our previous work also confirmed that the durability of analog switches can be further extended to ~10

By adjusting the switch conductance range (

). Then, we can roughly estimate that our memristor can support continuous neural signal processing for at least 5 years (for details on estimation, see Materials and Methods). For most typical applications, this lifetime is sufficient, and the lifetime can be further extended by using a memristor array with a larger number of columns.

) The output correlation matrix Δ

In different trials. The interictal and pre-ictal signals were used for the first 100 trials and the last 100 trials, respectively. (

) The shaded error bar graph of Δ

The waveform of the corresponding signal. (

) Distribution of Δ

The difference between interval and period trials. (

)Δ

The picture is a map drawn from different experiments, where the same suite and suite neural signal sets are applied to two different memristor arrays. The path from top to bottom and left to right is labeled T

To T

Correspondingly. (

) Correlation matrix between different experiments. T

And T

: Apply the same interval signal segment to the experiments of array #1 and #2 respectively. T

: Apply the same assertion signal clip to the trials of array #1 and #2, respectively.

Please note that there are differences in experimental results between experiments, which are mainly due to the periodic changes between memristors (

).

Display Δ from test to test

The waveform of the processing result.

The changes in the interocular signal and the frontal signal show similar distributions under small SD (3.74μS vs. 3.93μS). along with

, Which indicates that the difference between trials is very small, and therefore does not prevent the system from distinguishing between interictal and pre-ictal signals. In addition, the differences between the devices in the memristor array may also affect system performance. To study this effect, we processed the same neural signal segment on 16 channels of the memristor array, as shown in Figure 2. S6. The results show that the different channels in the memristor array show similar output Δ

Graph of the same input signal (Figure S6, E and F). In order to further verify the spatial scale of the memristor used for multi-channel processing of neural signals, that is, the robustness of the array level, two sets of new 16-channel iEEGs were processed on two different memristor arrays, and the results were summarized as follows:

. Similarly, it can be seen from the correlation diagram that the room and room signals can be distinguished. Future equipment optimization to reduce the variability of the memristor array is expected to further improve system performance.

After processing the input signal on the memristor array, the conductance (or equivalent Δ

It is possible to read the same initial conductivity value of all memristor devices) for further analysis, such as feature extraction and classification, which can be done by general machine learning algorithms. For proof of concept, here we can simply use the number of devices in a specific Δ

The range can justify the prediction of seizures. Reasons for the number of devices with a specific Δ

The input as the classification is the processed result, namely Δ

The distribution can represent signal energy and variation, as described above. as the picture shows

, The distribution of Δ

Although the change has similar signal energy, it can be measured by the periodic change that the signal from the periodic signal is more scattered than the signal from the periodic signal.

) Average Δ

Relative to the signal energy of eight consecutive segments. (

The relationship with the signal changes of eight consecutive segments. (

) The training results of the LDA classifier for 54 16-channel signal segments and the test results of the trained LDA classifier for 6 16-channel signal segments are plotted. F

, F

And F

Use Δ to indicate the number of equipment

Respectively in the specific range of (-35μS, -27μS), (-16μS, -9μS) and (-7μS, 0μS). (

) Training and testing accuracy from 10-fold cross-validation.

In order to prove the feasibility of using the proposed method to predict seizures, as described above, 60 sets of pre-seizure and inter-seizure iEEG signals were processed in our memristor array. Number of devices with Δ

∈ (-35 microseconds, -27 microseconds) (F

),Δ

∈(−16μS,−9μS) (F

) And Δ

∈(−7μS, 0μS) (F

) Is used as the extracted feature. Use linear discriminant analysis (LDA) to design a linear classifier (

) To distinguish the room and the signal in front of the room. To be fair, we use 10-fold cross-validation to train and test the classifier.

The results of 1 out of 10 trials are shown. The average accuracy of the 10 training and test trials were 95.33 and 95.00% (

), just use simple linear classifiers, which can be easily implemented in hardware. The average sensitivity and specificity of all test tracks are 93.33 and 96.67% respectively (for detailed calculations, please refer to Materials and Methods). It should be noted that more features can be extracted from the processing results of multi-channel neural signals to process more complex data sets and achieve higher classification accuracy.

Finally, the performance of the memristor-based hardware system is compared with the related literature work using memristors, and the performance of complementary metal oxide semiconductor (CMOS) application specific integrated circuit (ASIC) chips for neural signal processing is also compared. Benchmarks. This comparison is compared in detail in "Materials and Methods". The data and methods are used to process a segment of 16-channel neural signal segment with a sampling rate of 10 kHz (duration about 1-s). Compared with the CMOS ASIC reported in the literature about 80μW per channel, the power efficiency of 60.81 nW per channel in this work shows an improvement of about three orders of magnitude. Therefore, our system shows substantial advantages in the parallel processing of low-power multi-channel neural signals.

In addition to power consumption advantages and high scalability, our memristor array-based system also has the potential for real-time processing by using two memristor arrays to alternate conductance modulation and reading steps. Similarly, in our method, the input signal is segmented and directly applied to the corresponding memristor, essentially eliminating the need for a data buffer for temporarily storing neural signal segments. In addition, our system performs a lot of calculations and compresses the results into device conductance, thereby reducing the need for complex circuit modules for calculations and additional storage of intermediate and final data. These advantages can help significantly reduce the chip area, which is also a serious challenge for implantable biomedical electronics.

In summary, we have designed a highly scalable low-power hardware system for parallel processing of multi-channel neural signals based on memristor arrays. In addition, based on the results of experimental processing, it has been proved that the accuracy of epileptic seizure prediction is as high as 95% or more to verify the feasibility and efficiency of the system. The analog conductance modulation behavior of the memristor allows us to extract the critical amplitude and change information of the input signal and encode it in the conductance change of the memristor. With the signal segmentation scheme developed in the memristor array, the system has demonstrated excellent robustness and resistance to device changes. Our work highlights the huge potential of using memristor arrays to process multi-channel neural signals in parallel with high fidelity and power efficiency, which is essential for future fully implanted neural interfaces.

In the 1T1R structure, the transistor is used as a selector, and the resistive switch layer is fabricated on top of the drain terminal of the transistor. For our 1k cell memristor array, the transistor array is manufactured by a commercial foundry in a 0.13μm CMOS process. The remaining process steps are completed in the laboratory. For more detailed information about equipment manufacturing, please refer to our previous work (

).

We illustrate our detailed system overview in Figure 5. S1. In this picture, we propose a

X

A memristor array with peripheral circuits, where

Represents the number of channels and segments respectively. The design supports four operating modes, including FORM, SET, PROCESS and READ. The first three modes are used for electroforming, setting and resetting the memristor. In this work, the PROCESS mode is selected to process multi-channel neural signals in parallel through the RESET operation. In the reading mode, a current sense amplifier (CSA) with a resolution of 3 bits is used as the reading circuit. Figure S7 illustrates the programming scheme in PROCESS mode. Activated BLs and modules used to process specific parts of the neural signal are color-coded, while those that are not used are drawn in gray. Parallelism can be achieved in this mode by applying multi-channel input signals to the memristors in a column (ie segment) to simultaneously modulate their conductance. The shift register controls and selects which memristor column the input signal is applied to, and only one WL is turned on at a time. Similarly, figure. S8 shows the reading scheme of the memristor array in the reading mode. When sensing and digitizing the current, the CSA clamps the SL voltage to 0.15 V (ie, the read voltage), and then compares the current with the reference current. Figure S9 shows the FORM and SET schemes of the array. In these two modes, the memristor can be programmed in parallel or sequentially.

The 16-channel neural signal comes from the widely used open-access data set Kaggle seizure prediction data set (

). The data set includes multi-channel neural signals from four dogs and two human patients. In our work, a neural signal sampled from "dog 2" at 400 Hz is used. We randomly selected 60 groups of onset and onset signals for classification. Here, the pre-seizure signal refers to a brain signal segment recorded tens of minutes before the onset of the epileptic seizure, and the interictal signal was recorded from the brain at least 1 week before the next seizure occurred.

Correlation coefficient, use

It is a statistic used to measure the linear relationship between two sets of data. the value of

Between -1 and 1. The closer its absolute value is to 1, the better representation of these two data sets can be achieved by linear equations. There is almost no linear relationship between them, or

Close to 0. For any given two vectors

Have the same length

, Their correlation coefficient

It can be calculated using the following formula

where

Yes average

respectively.

Figure S5 shows that the typical durability of our memristor can reach more than 2×10

cycle. Because the memristor has such a long service life, we did not observe any significant drop in system accuracy in the experiment. In this work, we use the neural signal sampled at 400 Hz as the input of the memristor array, so when processing one segment (64 segments) of the neural signal segment, each device has to withstand 15 RESET voltage pulses (Ie a fragment). About 2.4 seconds. Then, we can roughly estimate this endurance as 2×10

The loop can continuously process 2×10 neural signals

/ 30×2.4 seconds = 1.6×10

s> 5 years, assuming that the same number of SET pulses (15) are needed to initialize the memristor conductance after each segment. Therefore, the lifetime of the memristor array is sufficient to meet the needs of most applications.

In the prediction, the early samples are considered positive samples, and the interphase samples are considered negative samples. The results of the LDA classifier can be divided into four categories: true positive, false negative, true negative and false negative. Here, "true" and "false" indicate correct and incorrect predictions, respectively. In order to evaluate the prediction performance, three statistical indicators are calculated as follows, including accuracy, sensitivity and specificity.

To estimate the power efficiency, we used a task to process the 16-channel neural signal sampled at 10 kHz. The whole task can be divided into two parts: multi-channel signal processing, and then read the conductance state of the memristor. For comparison with CMOS ASIC, 16 consecutive input pulses are applied to each memristor, which is similar to the 15-pulse experiment in this demonstration. Then, a signal segment with 10,240 samples in each channel can be processed in a 16×640 memristor array to estimate power efficiency.

The module used for multi-channel signal processing steps is shown in Figure 2. S7. In order to handle the tasks defined above, the energy consumed in the array operation can be calculated as (2 V)

×30μS×50 ns / pulse×16×640×16 = 983.04 nJ, where 2 V is the upper limit of the input voltage amplitude, because the input voltage pulse is linearly transformed to the range of 1 to 2 V, as shown in Figure 2.

(C and D). Here, the initial memristor conductance is 30 μS, and the pulse width is 50 ns. The energy consumed by the peripheral circuit is calculated as follows: For 640 BL multiplexer (MUX), 0.6μW×50 ns/pulse×16×16 pulse×640 = 4.92 nJ; 0.6μW×50 ns/pulse×16×16 pulse ×For 16 SL MUXs, it is 640 = 4.92 nJ, where in PROCESS mode, the power consumption of BL / SL MUX is 0.6μW. In the programming scheme of PROCESS mode, the dynamic power consumption of the shift register is only negligible, which is estimated to be 10 kHz / 16×(5 V)

×10 fF×(10,240 samples / 10 kHz) = 0.16 nJ, where 5 V is the source voltage and 10 fF is the transistor gate capacitor. Therefore, the total energy of this step is estimated to be 993.04 nJ.

The module used for reading mode is shown in Figure 2. S8. During the above tasks, the energy consumed in the memristor device can be estimated as (0.15 V)

×30μS×50 ns/pulse×16×640×16 = 0.35 nJ, where 0.15 V is the reading voltage. The energy consumed by the peripheral circuit is calculated as follows: For 640 BL MUX, 0.045μW×50 ns/pulse×16 pulse×640×16 = 0.37 nJ and 0.045μW×50 ns/pulse×16 pulse×640×16 = 0.37 16 NJ of SL MUX, where 0.045μW is the power consumption of BL/SL MUX in READ mode. The 16 CSAs consume 4μW×50 ns×16×640 = 2.05 nJ of energy to sense and digitize the output conductance, among which the 4μW CSA power consumption is estimated through simulation. Similarly, the dynamic power consumption of the shift register is only 0.16 nJ. Therefore, the total energy for this step is 3.30 nJ.

To calculate the power efficiency, we first estimate the total energy consumption to be 996.34 nJ. Therefore, the power efficiency is estimated to be 996.34 nJ / (10,240 samples / 10 kHz) / 16 channels = 60.81 nW per channel. For similar products based on CMOS, (

) Reported an ASIC chip for epileptic seizure control, in which the biological signal processing unit consumes a total of 2.5 mW of energy to extract features based on 128-point fast Fourier transform, and perform entropy calculation based on 16-input ridge regression model for classification . Therefore, its power efficiency is calculated as 2.5 mW / 16 channels = 156.2μW/channel. Cheng

) Reported another ASIC chip with a similar power efficiency of 163.2μW per channel. According to the amount of calculation used for feature extraction and classification, it is estimated that the power consumed by the feature extraction of these two ASIC chips exceeds 50%. Therefore, compared with similar products based on CMOS, the neural signal processing system based on memristor shows more than 1000 times advantage in power efficiency (60.81 nW per channel versus ~80μW per channel).

For supplementary materials for this article, please visit:

.

This is an open access article distributed under the following terms

, It allows use, distribution and reproduction in any medium, as long as the final use is

For commercial interest, and provide the original works appropriately cited.

Volume 6, Number 41

October 7, 2020

Thank you for your interest in promoting the term "scientific progress".

Note: We only ask you to provide your email address so that the person you recommend to the page knows that you want them to see the email and that it is not spam. We will not capture any email addresses.

This question is used to test whether you are a visitor and prevent automatic spam submission.

Please log in to add an alert to this article.

by

: Eabc4797

The memristor array provides a scalable and efficient platform for the parallel processing of multi-channel neural signals.

Volume 371, Issue 6526

©2021

. all rights reserved. The American Association for the Advancement of Science is 

, 

 with 

ISSN 2375-2548.