Kimi K3 and Fable routing strategy achieve 93% accuracy, with costs potentially reduced by up to 50 times

Fireworks AI released an evaluation report comparing the open-source model Kimi K3 and the closed-source model Fable 5 across approximately 1030 agent tasks (covering scenarios such as SWE, terminal operations, algorithms, multilingual coding, and legal matters). The results showed that the accuracy rates for the two models on the SWE benchmark were 92.4% and 92.6%, respectively, indicating similar overall performance, but each model has its strengths in specific areas: K3 leads in symbolic mathematics and development tools, while Fable excels in web and data visualization; in long-cycle terminal tasks, K3 independently tackled 11 tasks that Fable could not solve.

The report pointed out that adopting a task-level routing strategy can dynamically allocate between the two, achieving an accuracy rate of 93%, surpassing either single model. Additionally, due to K3's significant cost advantage on the Fireworks platform, costs for long-cycle agent tasks can be up to 50 times lower than Fable. The report recommends using the open-source model as the default option and continuously learning the optimal match between tasks and models through routers to balance quality and cost.