Avalanche Roadmap to Beta-2 🚀

[vlomonaco]

Here we report a draft roadmap as an ordered list by features priority and divided by main modules. This roadmap targets Avalanche Beta 2 Release (v0.2.0) set for the 31st of July 2022.

Bugs are always solved first with maximum priority. New features from the "Feature Request" list may be added, but this roadmap keeps track only of the open issues already planned to be integrated for the Beta Release! 🚀

Benchmarks

• iNaturalist large-scale dataset implementation iNaturalist large-scale dataset implementation #298
• Mirroring OpenLoris and Add new link for Auto-Download Mirroring OpenLoris and Add new link for Auto-Download #617
• Uniform API for classic Benchmarks Uniform API for classic Benchmarks #557
• Change "Scenario" in "Benchmark" Object Change "Scenario" in "Benchmark" Object #503
• Fallback mechanism when "targets" is not found Fallback mechanism when "targets" is not found #422
• Add Doc + Utils to Support Multiple Streams for Benchmark Add Doc + Utils to Support Multiple Streams for Benchmark #519
• Add Pascal VOC2012 Classic Benchmark Add Pascal VOC2012 Classic Benchmark #536
• Lazy definition of benchmarks Lazy definition of benchmarks #600
• Remove task labels for SIT Remove task labels for SIT #464
• CORe50 Dataset and Benchmark Improvements CORe50 Dataset and Benchmark Improvements #547
• Integrating the CtRL Benchmark add CTrL integration #561
• Add Task Descriptors, not only Task Labels Add Task Descriptors, not only Task Labels #371
• Unbalanced Steps Sizes Option for Benchmarks Generators Unbalanced Steps Sizes Option for Benchmarks Generators #331
• Add MiniImagenet to the Benchmarks Add MiniImagenet to the Benchmarks #154

Evaluation & Logging

• Confusion Matrix for W&B Evaluation Confusion Matrix for W&B #564
• Add summary table [Metrics Name, Brief Description] Add summary table [Metrics Name, Brief Description] #585
• Extend metrics to work with task label at patterns level Extend metrics to work with task label at patterns level #647
• Add TestPlugin to test metrics and other features within callbacks Add TestPlugin to test metrics and other features within callbacks #498
• Add (Lopez-paz, 2017) Metrics: ACC, FWT & BWT Add (Lopez-paz, 2017) Metrics: ACC, FWT & BWT #212

Models & training

• Strategies: Standardization of Initial Weights (SIW) Plugin implementation #527 Add YASS baseline #274 Add ICaRL to the CL Baselines #4 Add Progressive network strategy #85 BiC (@mmasana) (to be updated). New strategies already implemented and tested: ICaRL, PNN, several replay policies.
• Update Strategy&Plugin API: see discussion in Improvement to BaseStrategy #580 Incremental classification strategy and best practices #329 Inputs to the forward- and criterion-functions in the BaseStrategy-object are limited #596 Override some training parameters in `train` call #563 Modifying a strategy to allow for the model to have more than one output. #437 Module Independency & Standalone Usability #400 Need for Explicit Assumptions #394 Change "Scenario" in "Benchmark" Object #503
• Update Documentation (after the API update)

Continuous Integration & Tests

• Push notebooks on Colab Port notebooks on Colab and Sync them with Gitbook #381
• Fix dependencies version Fix dependencies version #611
• Add Automatic Docker Release Add Automatic Docker Image Release #383
• Conda & Pi packages Create Public Conda and Pip Packages #135
• Automatic GPU/Deps/Extended tests on VPS Automatic GPU Tests on VPS #620

Documentation

• Expand FZTH "models" chapter Expand FZTH "models" chapter #571
• Custom MetricPlugin example Custom MetricPlugin example #325
• How-To Tutorials on Gitbook documentation How-To Tutorials on Gitbook #568
• Check Offered Functionalities <-> Doc sync Check Offered Functionalities <-> Doc sync #558
• Overall Update of Avalanche Website

[AndreaCossu]

Don't we already have the beta project here on github for this purpose?

[vlomonaco]

I think it is easier to have a single issue summarizing everything. In the end we don't really use the features offered in the "projects" tab. If this works I will close down the projects there but I'm open to suggestions of course.

[AntonioCarta]

I agree with @vlomonaco. Roadmap issues are better and they allow for more discussion. I think it's better to close the project to avoid duplication and inconsistent roadmaps.

[AndreaCossu]

As long as we keep a single place I'm fine with both. I don't think we would be able to keep the two aligned.

[AntonioCarta]

I added some issues about the training API that I would like to tackle before the beta release.

Overall, these is a brief recap of the issues/feedback from the users that I collected over time:

Requests

• need a simple way to define/customize new strategies [BACKWARD INCOMPATIBLE] Explicit return values for strategy methods #660
• before/after training semantic [BACKWARD INCOMPATIBLE] Behavior when training on multiple experiences #661
• scenarios' constraints
• some methods are part plugin and part strategy (or more than one of each), it can become more difficult to understand its implementation. The lack of understanding of how some plugins or strategies work can be detrimental since someone can be bypassing the rules/assumptions of some scenarios. [BACKWARD INCOMPATIBLE] Plugins and interactions with strategy and arguments #659
• having one clear way to implement strategies would always make things easier.
• Having metrics and loggers into plugins makes sense with the name plugins, but having them mixed with the methods and strategies is quite odd to me
• some plugins are not compatible between them.

CANT/WONT fix

• experience/scenario reference: This is in direct contrast with the request for scenarios' constraints and information hiding. Also, at the moment the scenario can be retrieved already from the experience.

• module independence: Depending on how you interpret this issue, it's either already solved or unsolvable.

• override train params (train_epochs, mb_size, ...) Solved, but in a different way. We can use overriding of strategy methods. Having a single, easy way to customize strategies seems easier than having multiple ways that may be slightly broken when mixed together. See [BACKWARD INCOMPATIBLE] Plugins and interactions with strategy and arguments #659

• custom forward, loss <- these are already implemented