Hi! My research collaborators and I happened to be in possession of a significant amount of compute. We’re inspired by the Eleuther’s team Pythia repo and would like to make a similar model suite for interpretability/alignment research (models at varying levels of sizes, with ~150 weight checkpoints, training data, and relevant hyperparameters, like training schedule).
I’m curious where there is demand here to answer mechanistic interpretability/scaling research questions. Some options are:
ViT models
Other language models (e.g. opensource BLOOM)
Multimodal models (e.g. video models like Phenaki)
Call for Pythia-style foundation model suite for alignment research
Hi! My research collaborators and I happened to be in possession of a significant amount of compute. We’re inspired by the Eleuther’s team Pythia repo and would like to make a similar model suite for interpretability/alignment research (models at varying levels of sizes, with ~150 weight checkpoints, training data, and relevant hyperparameters, like training schedule).
I’m curious where there is demand here to answer mechanistic interpretability/scaling research questions. Some options are:
ViT models
Other language models (e.g. opensource BLOOM)
Multimodal models (e.g. video models like Phenaki)
Thank you in advance.