CoFrGeNets replace the ‘bones’ of transformer-based models

IBM Research · 1 week ago

IBM Research developed CoFrGeNets, a new neural architecture that replaces transformer components with structures based on continued fractions to reduce parameters and computational overhead. CoFrGeNet models matched or outperformed GPT2-xl and Llama-3.2B while using hundreds of millions fewer parameters and training faster. This architectural approach could be selectively integrated into existing model pipelines to improve efficiency without sacrificing performance.