• 0 Posts
  • 63 Comments
Joined 1 year ago
cake
Cake day: June 23rd, 2023

help-circle


  • You say “Not even close.” in response to the suggestion that Apple’s research can be used to improve benchmarks for AI performance, but then later say the article talks about how we might need different approaches to achieve reasoning.

    Now, mind you - achieving reasoning can only happen if the model is accurate and works well. And to have a good model, you must have good benchmarks.

    Not to belabor the point, but here’s what the article and study says:

    The article talks at length about the reliance on a standardized set of questions - GSM8K, and how the questions themselves may have made their way into the training data. It notes that modifying the questions dynamically leads to decreases in performance of the tested models, even if the complexity of the problem to be solved has not gone up.

    The third sentence of the paper (Abstract section) says this “While the performance of LLMs on GSM8K has significantly improved in recent years, it remains unclear whether their mathematical reasoning capabilities have genuinely advanced, raising questions about the reliability of the reported metrics.” The rest of the abstract goes on to discuss (paraphrased in layman’s terms) that LLM’s are ‘studying for the test’ and not generally achieving real reasoning capabilities.

    By presenting their methodology - dynamically changing the evaluation criteria to reduce data pollution and require models be capable of eliminating red herrings - the Apple researchers are offering a possible way benchmarking can be improved.
    Which is what the person you replied to stated.

    The commenter is fairly close, it seems.


  • Monument@lemmy.sdf.orgtoMemes@lemmy.mlToxicity
    link
    fedilink
    English
    arrow-up
    3
    ·
    26 days ago

    That’s very fair, indeed.

    Perhaps awareness of one will spark awareness of the other. I suppose my concern is that plasticisers are sort of a ‘hidden’ risk, for the most part. They’re used in nearly every food packaging (and prep, such as hoses) that isn’t contained in glass, or served up in its own peel.


  • Monument@lemmy.sdf.orgtoMemes@lemmy.mlToxicity
    link
    fedilink
    English
    arrow-up
    41
    ·
    27 days ago

    Microplastics are terrifying and all that, but I’m sort of more worried about plasticisers like BPA, BPF, BPS and the rest of the alphabet of BP-whatever’s that was created and brought into use after the dangers of BPA were realized.

    Just a heads up - if something plastic says it’s BPA-free, it probably uses a different bisphenol compound that is less studied than BPA. And is likely as toxic (or even more toxic)!

    But nobody ever talks about those, because science words.













  • Honestly kind of excited for the company blogs to start spitting out their disaster recovery crisis management stories.

    I mean - this is just a giant test of disaster recovery crisis management plans. And while there are absolutely real-world consequences to this, the fix almost seems scriptable.

    If a company uses IPMI (Called Branded AMT and sometimes vPro by Intel), and their network is intact/the devices are on their network, they ought to be able to remotely address this.
    But that’s obviously predicated on them having already deployed/configured the tools.