Conclusion
A number of bigger conclusions emerge from this take a look at case. The 2 fashions that drew from curated databases of experimental literature, NotebookLM and our custom-built device, outperformed the LLMs skilled on unfiltered web information. Specifically, fashions counting on open internet sources tended to combine established theories with extremely speculative ones.
The evaluated LLMs (accessed in December 2024) additionally confirmed weaknesses in temporal and contextual understanding. For instance, they typically failed to acknowledge when a proposed speculation was later disproved. In addition they often omitted related papers once they didn’t explicitly embrace the precise language used within the preliminary question.
Our outcomes broadly spotlight the necessity for LLMs to raised perceive tables and pictures, as scientific papers closely use these codecs. Whereas two of the fashions constantly referenced photographs, they typically relied extra on picture captions fairly than on visible evaluation. Enhancing visible reasoning functionality, together with deciphering photographs, plots and scale bars, is a significant path for future enchancment.