🧠 Project Progress & Research Journey

From a Broad Survey of Continual Learning to a Novel Project Hypothesis

Student: Abhinav
Github: https://github.com/letsbegincode/Incremental-Learning/
Date: September 11, 2025
📚 Phase 1: Understanding the Landscape
❗The Core Problem
The primary challenge in this field is Catastrophic Forgetting, where deep neural networks forget previously learned knowledge upon learning new tasks.
🎯The Goal
The objective is to achieve Continual Learning: enabling models to learn sequentially from a stream of data, accumulating knowledge over time.
Key Insight: A foundational understanding was built upon comprehensive survey papers that map out the key challenges and the primary families of solutions.
⬇️
🧭 Phase 2: Evaluating Foundational Strategies
⚖️Regularization
Methods like EWC protect important weights from previous tasks.
Verdict: Promising but Limited. Research (e.g., van de Ven & Tolias, 2019) shows these methods often struggle in challenging Class-Incremental scenarios where task boundaries are blurry.
📚Rehearsal
Storing and replaying a small subset of past data (exemplars).
Verdict: Most Effective Baseline. This approach has consistently shown strong performance and robustness, making it a powerful and practical foundation to build upon.
🏗️Architectural
Dynamically expanding the network for new tasks.
Verdict: Impractical for Project Scope. While effective, these methods often introduce significant complexity and memory overhead, making them less suitable for this project.
⬇️
🔍 Phase 3: Identifying a Gap in the Strongest Approach

🤔 Critical Observation on Rehearsal Methods (like iCaRL)

After selecting Rehearsal as the most promising strategy and studying its state-of-the-art implementation in papers like iCaRL (Rebuffi et al., 2017), a key limitation was identified:

The Inefficiency of Constant Rehearsal: Existing methods employ a *static* rehearsal strategy. They constantly replay exemplars with a fixed intensity, regardless of whether the model is actually forgetting. This is computationally wasteful, analogous to studying flashcards for a subject you already know perfectly.

⬇️
💡 Phase 4: Formulating a Novel Hypothesis

⚡ The "Aha!" Moment: Connecting Two Fields

The identified gap (inefficient rehearsal) led to a crucial question: "How can a system know *when* it's starting to forget?"

The answer lies in a different but related field: **Stream Mining**. This field has developed robust algorithms for **Concept Drift Detection**—methods designed to detect when the statistical properties of a data stream change. By re-framing "forgetting" as a drop in performance on past tasks, we can see it as a form of concept drift.

⬇️

🎉 Proposed Innovation: "Smart Rehearsal"

Project Hypothesis: By integrating a formal concept drift detector (ADWIN) to monitor model performance on past tasks, we can create an adaptive "Smart Rehearsal" system. This system will trigger intensive rehearsal only when forgetting is detected, achieving comparable accuracy to standard rehearsal methods but with significantly lower computational cost.

iCaRL Rehearsal Core

Provides the strong, proven baseline for storing and replaying exemplars.

Concept Drift Detector

Acts as an efficient, real-time "forgetting alarm" by monitoring performance.

Adaptive Trigger

Connects the two components to initiate "rehearsal bursts" only when necessary.

🗺️ Project Timeline & Deliverables

  1. 1
    Foundation & Baseline (Sept - Oct 2025): Master foundational papers and implement a standard iCaRL model as a performance baseline.
  2. 2
    Hybrid Model Development (Nov 2025): Integrate the ADWIN drift detector from the `river` library into the iCaRL framework to create the "Smart Rehearsal" system.
  3. 3
    Experimentation & Analysis (Dec 2025): Conduct comparative experiments on standard benchmarks (e.g., Split CIFAR-10) to measure both accuracy and computational efficiency.
  4. 4
    Final Report & Presentation (Jan 2026): Document the methodology, analyze results, and present the findings, highlighting the efficiency gains of the proposed model.

📚 Key References

  • [1] Rebuffi, S. A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017). iCaRL: Incremental Classifier and Representation Learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR).
  • [2] van de Ven, G. M., & Tolias, A. S. (2019). Three scenarios for continual learning. arXiv preprint arXiv:1904.07734.
  • [3] Bifet, A., & Gavaldà, R. (2007). Learning from time-changing data with adaptive windowing. In Proceedings of the 2007 SIAM international conference on data mining. (This is the original paper for ADWIN).
  • [4] Montiel, J., Read, J., Bifet, A., & Abdessalem, T. (2021). River: machine learning for streaming data in Python. Journal of Machine Learning Research.