Google faces mounting evidence of its search monopoly. A federal judge recently ruled – for the second time in a year – that Google illegally maintains a monopoly in ad tech. Google's response?
The United States set out to win a trade war with China. It failed. Years of tariffs, tech restrictions, and tough rhetoric have backfired, leaving America in a weaker position while China charges ahead with innovation and global influence.
Good Morning from San Francisco, Meta just kicked Apple's AI tools to the curb. All of them. 🥾 No
Even the most advanced AI visual systems have a serious problem: They try to answer questions they can't actually solve. A new study from the University of Tokyo and their collaborators tested leading AI models on what seems like a simple task - knowing when to say "I can't answer that."
The results revealed a concerning gap between what these systems claim to understand and what they truly comprehend. The researchers created three types of impossible questions. They removed correct answers from multiple choice options, showed images that had nothing to do with the questions being asked, or provided completely irrelevant answer choices. A reliable AI system should recognize these situations and decline to answer.
But that's not what happened. While these same AI models score impressively on standard tests, they performed dismally when faced with impossible questions. Many open-source models got scores below 6% when they should have said "I can't answer this."
The gap between closed-source models (like GPT-4 Vision) and open-source alternatives proved particularly stark. While GPT-4 Vision managed to identify unsolvable questions about 60% of the time, popular open-source models like CogVLM2 scored below 1% - despite both performing similarly well on standard tests.
"This suggests that our community's efforts to improve performance on existing benchmarks do not directly contribute to enhancing model reliability," the researchers note. In other words, we've been teaching AI to guess even when it shouldn't.
The study uncovers different failure patterns among models. Some struggle specifically with visual tasks, while others have trouble with basic reasoning about whether questions are answerable. The researchers found that adding explicit instructions to consider whether questions were impossible helped some models but made others perform worse.
Looking ahead, the team suggests that future AI development needs to focus not just on getting right answers, but on knowing when getting an answer isn't possible.
Why this matters:
Read on, my dear:
A new artificial intelligence system from China's Shandong First Medical University helps scientists understand how genes turn on and off. Called TRAPT, it maps gene control with record-breaking accuracy.
The White House's new science chief wants to overhaul U.S. research funding. In an interview with Bloomberg News, Michael Kratsios laid out his vision for smarter spending on technology research despite sweeping budget cuts.
InternVL3, a new open-source AI model, matches or beats proprietary giants like GPT-4 and Gemini in understanding images and video. Its key breakthrough lies in how it learns to process visual information alongside language from the start, rather than adding these capabilities later.
Stanford and Google DeepMind researchers built an AI system that can analyze millions of street photos to track how cities change over time. The system found striking patterns.
Fuel your morning with AI insights. Lands in your inbox 6 a.m. PST daily. Grab it free now! 🚀
