An organization working on math benchmarks for AI faced accusations of impropriety for not disclosing funding from OpenAI until recently.
Epoch AI, a nonprofit mostly funded by Open Philanthropy, revealed on December 20 that OpenAI had supported the creation of FrontierMath. This test, designed to measure an AI’s mathematical skills, was used by OpenAI to showcase their upcoming flagship AI, o3.
In a forum post on LessWrong, a contractor for Epoch AI known as “Meemi” expressed concerns about contributors not being informed of OpenAI’s involvement until later.
Online users raised worries about the lack of transparency affecting FrontierMath’s reputation as an impartial benchmark due to OpenAI having access to the problems and solutions before o3’s announcement on December 20.
Epoch AI’s associate director acknowledged the lack of transparency and admitted to not negotiating stronger transparency terms with OpenAI. However, they assured that OpenAI has a verbal agreement not to use FrontierMath for AI training.
Despite assurances, Epoch AI has not been able to independently verify OpenAI’s FrontierMath o3 results, highlighting the challenge of creating unbiased AI benchmarks without perceived conflicts of interest.
