Intelligence Modalities
In the eFabric™ ecosystem, "Intelligence" is not a one-size-fits-all concept. Different sensors produce different data patterns, each requiring a specific mathematical approach. While eFabric™ is world-renowned for its Audio Intelligence, the platform is architected to be a multi-modal factory, extending into Sensor Fusion and Vision AI.
Audio Intelligence: Keyword Spotting vs. Sound Identification
Audio is the primary modality for "Always-On" devices. eFabric™ specializes in processing acoustic data at the Microwatt Scale, allowing devices to listen for specific signatures without invading user privacy or draining battery life. In the eFabric™ ecosystem, audio isn't just a sound wave; it is a complex, time-varying signal that must be compressed and analyzed within a 100ms - 200ms latency window.
The Audio Pre-processing Pipeline
Before the Neural Network "hears" anything, the raw PCM (Pulse Code Modulation) audio must be transformed. eFabric™ automates this via a sophisticated DSP pipeline:
- Framing & Windowing: The continuous stream is sliced into small "frames" (typically 20-40ms). We apply a Hamming or Hann Window to these frames to minimize spectral leakage at the edges.
- The Log-Mel Spectrogram: This is the "Vision of Sound." We convert the time-domain signal into the frequency domain using a Fast Fourier Transform (FFT). Link We then map these frequencies to the Mel Scale, which mimics human hearing by prioritizing lower frequencies where speech and most mechanical "signatures" reside.
- Logarithmic Scaling: We apply a log function to the energy levels. This ensures the model can detect a "Whisper" just as accurately as a "Shout," providing the Dynamic Range necessary for real-world robustness.
Keyword Spotting (KWS):
KWS is the most demanding audio modality because it requires a 0.0% False Rejection Rate (FRR) in quiet environments and a minimal False Acceptance Rate (FAR) Linkin noisy ones.
- The Concept: Detecting a specific "Wake Word" (e.g., "Hey Meritech").
- Feature Buffering: eFabric™ uses a sliding window (Circular Buffer) to hold the last 1–2 seconds of audio features.
- The Technology: Uses a CNN-based architecture to monitor a continuous stream of audio. It looks for a high-confidence match against a pre-trained Spectrogram signature. This means the model learns the transition from one sound to the next, making it resistant to different accents and speaking speeds.
- Optimization: eFabric™ models are trained to be robust against "Noise Injection"—they can hear the keyword even in a crowded room or a moving vehicle. To prevent false triggers from the TV or radio, eFabric™ allows you to train a "garbage" or "shadow" class—a model that learns what isn't your keyword but sounds similar.
Sound Event Detection (SED):
While KWS looks for specific patterns, SED identifies Acoustic Signatures.
The Concept: Identifying non-speech sounds that signify an event.
The Use Cases: Glass Break Detection, Baby Cry Monitoring or Emergency Siren Identification.
Technical Edge: Unlike KWS, SED often requires the model to understand the duration and intensity of a sound, frequently utilizing RNN/GRU layers to track the temporal "shape" of the noise.
Spectral Complexity: Sounds like Glass Breaking or Alarms have high-frequency transients and wide spectral spreads. eFabric™ uses specialized kernels in its CNN layers to identify these "percussive" events.
Temporal Memory (GRU Integration): For events that take time to unfold (like a Baby Crying), eFabric™ employs Gated Recurrent Units (GRUs). This allows the model to "remember" the rhythmic nature of the cry over several seconds, distinguishing it from a single loud noise.
The Developer’s Technical Checklist for Audio
| Metric | Target Specification | eFabric™ Advantage |
|---|---|---|
| Sample Rate | 16 kHz (Standard) | Optimized for the Syntiant® NDP Audio Buffer. |
| Bit Depth | 16-bit PCM | Direct ingestion; no external transcoding needed. |
| Inference Latency | < 200 ms | Silicon-native path ensures near-real-time response. |
| Memory Footprint | < 100 KB (Weights) | Achieved via Int8 Quantization and DSP pruning. |
Sensor Fusion & Time-Series Data
While audio captures the "acoustic signature" of an environment, Sensor Fusion focuses on the physical dynamics of the world. In the eFabric™ ecosystem, this involves processing high-frequency data from Inertial Measurement Units (IMUs)—specifically accelerometers and gyroscopes—and environmental sensors. This is known as Time-Series Analysis.
For developers, the challenge here is Temporal Correlation: identifying how a sequence of physical movements over time constitutes a specific event.
A. The Time-Series Processing Engine
Unlike audio, which is processed as a 2D spectrogram, sensor data is often treated as Multi-Channel Time-Series signals. eFabric™ employs a specialized pipeline to handle this:
Axes Synchronization: eFabric™ aligns data from 3-axis (X, Y, Z) accelerometers to ensure the neural network sees a unified "motion vector."
Signal Filtering: Raw sensor data is notoriously noisy. eFabric™ utilizes built-in Low-Pass and Band-Pass Filters to remove gravitational bias and high-frequency jitter before the data reaches the model.
Windowing & Overlap: To capture a "Double Tap" or a "Gesture," eFabric™ slices the sensor stream into windows. By using Sliding Windows with Overlap (e.g., 500ms windows with 50% overlap), the platform ensures that no event is lost between the cracks of two processing cycles.
B. Industrial Anomaly Detection: Predictive Maintenance
Vibration Fingerprinting: Every motor has a unique vibration "fingerprint." eFabric™ models are trained to recognize the Fundamental Frequency of healthy operation.
Harmonic Analysis: When a bearing wears down or a shaft misaligns, the sensor picks up "Harmonics"—secondary vibrations at specific intervals. eFabric™ uses CNN layers to detect these spectral changes in the vibration data.
Trend Analysis: By deploying these models on the TML120, factories move from "Scheduled Maintenance" to "Condition-Based Maintenance," saving thousands in downtime.
C. Human Activity Recognition (HAR) & Gestures
The "Memory" Factor (RNN/GRU): Detecting a "Fall" or a "Wrist Tilt" requires the model to understand the progression of movement. eFabric™ uses Gated Recurrent Units (GRUs) to maintain a Hidden State:
allowing the model to distinguish between a user sitting down and a user actually falling.
Gesture Logic: eFabric™ allows you to define "Active" vs. "Passive" motion states, ensuring the high-power application processor only wakes up when a specific, validated gesture is detected on the microwatt-scale NDP.
D. Automotive Intelligence: The EV Battery Sentinel
RUL & SOH Estimation: By analyzing microscopic fluctuations in temperature and electrical load (Time-Series data), eFabric™ models estimate Remaining Useful Life (RUL) and State of Health (SOH).
Thermal Stress Detection: The model monitors for "Thermal Runaway" signatures—patterns of heat and vibration that occur seconds before a battery failure—enabling immediate safety triggers.
The Developer’s Technical Checklist for Sensors
| Metric | Target Specification | eFabric™ Advantage |
|---|---|---|
| Sampling Rate | 50Hz to 1600Hz | Fully configurable to match the Nyquist frequency of the target event. |
| Data Channels | Up to 6 (Accel + Gyro) | Native support for multi-axis fusion without external math libraries. |
| Feature Extraction | RMS, Mean, Std Dev, FFT | Automated feature engineering optimized for Int8 Quantization. |
| Integration | I2C / SPI to NDPLink | Direct sensor-to-chip path; the host MCU remains in deep sleep. |
Vision AI
Vision at the edge has traditionally been a "power-hungry" modality, requiring high-bandwidth cameras and heat-generating processors. eFabric™ is disrupting this by bringing Computer Vision (CV) to the Syntiant® NDP200 and NDP250 Linkseries, enabling battery-powered devices to "see" and "interpret" their surroundings without ever leaving the microwatt power envelope.
A. The Shift to Low-Resolution Intelligence
In the eFabric™ ecosystem, Vision AI is not about high-definition recording; it is about Semantic Understanding.
The Concept: Instead of processing 1080p video streams, eFabric™ models operate on low-resolution, low-frame-rate visual data. The goal is to detect events, not to capture detail.
Privacy-by-Design: Because the model runs entirely on the NDP silicon, the raw image data is never stored or transmitted. The chip only outputs a high-level metadata trigger (e.g., Person_Detected = True), making it high standard for privacy-compliant smart spaces.
B. Key Vision Modalities
Binary Presence Detection & People Counting
Unlike high-power security cameras, eFabric™ Vision is designed for Binary Presence Detection. It answers the question: "Is a human in the frame?"
Face Recognition & Authentication
Implementing low-resolution "Face-ID" for secure entry points, where the model only activates high-power systems after a positive visual match is made locally.
Object Classification
Because the data is processed at the Silicon-Level, no images are ever stored or transmitted. The chip only outputs a "Metadata Trigger" (e.g., Person_Detected = True).
C. The Vision Neural Engine: NDP200/250
Parallel Windowing: The hardware can look at multiple parts of an image simultaneously to find a match, maintaining Deterministic Inference even with complex visual inputs.
Native Image Scaling: The NDP hardware automatically downsamples and normalizes camera input, offloading the heavy "image preparation" work from the neural network layers.
The Developer’s Technical Checklist for Vision
| Metric | Target Specification | eFabric™ Advantage |
|---|---|---|
| Input Resolution | Up to QVGA (320 x 240) | Optimized for feature extraction, not video playback. |
| Power Profile | < 5 mW during active vision | 100x lower than traditional embedded vision CPUs. |
| Model Type | Optimized MobileNet / SSD | Hardware-native mapping of 2D Convolutional layers. |
| Frame Rate | 1 to 5 FPS (Adjustable) | Sufficient for presence and state detection at the edge. |