Reverse Engineering the Response to "best smartphones 2025" in Chatgpt

       By Chris Byrne (6.8.25)

    This is my attempt to reverse engineer a response to a comparative entity prompt ("best smartphones 2025") in Chatgpt 4 using the tool itself. I chose this prompt to trigger "Retrieval Augmented Generation" (RAG): where instead of relying solely on the model’s pre-trained knowledge alone, the model retrieves relevant information from connected data sources (via Google search according to my interpretation of the feedback below). 

   In the USA for example, there are many thousands of "best smartphones"-type searches per month on Google alone, according to data from Google Keyword Planner - we might assume that LLMs might be used in a similar way to compare products etc. 

      See the original response to my prompt “Best Smartphones 2025” below:

“Based on current expert reviews and recent releases (as of August 2025), here are the top smartphones:

1. Samsung Galaxy S25 Ultra

• Snapdragon 8 Gen 4 / Exynos 2500 chip (region dependent)

• 200 MP camera, dual telephoto lenses with up to 100× zoom

• 6.8″ QHD+ AMOLED 120 Hz display

• AI features like “Circle to Search” and multitasking assistants

• 120 W wired / 50 W wireless charging

• Reviewed by TechAdvisor as the top Android flagship
  
  

2. iPhone 17 Pro Max

• Apple A19 Bionic chip with upgraded NPU

• ProMotion 120 Hz LTPO OLED display

• Deep integration with iOS 18's AI features

• Strong battery life and camera upgrades
  
  

3. Xiaomi 15 Ultra

• 50 MP quad cameras, Leica optics

• Snapdragon 8 Gen 4, 90 W fast charging

• Top-tier performance and image processing
  
  

4. OnePlus 13

• Snapdragon 8 Gen 4, 100 W charging

• Great value for performance

• Clean UI, 7 years of updates
  
  

5. Samsung Galaxy Z Fold 7

• Foldable flagship with a lighter hinge

• Snapdragon 8 Gen 4

• Multitasking UI with 8.1″ inner display

• Reviewed as best productivity phone”

My analysis (assisted by Chatgpt)

    Below based on using direct prompts I have attempted to show how the response to my prompt “best smartphones 2025” was created . I used the direct feedback from the tool itself. The below shows how Chatgpt used external web searches to supplement it's preexisting knowledge base while distinguishing between what info used in the response came from it's training in contrast with what came from up to date web sources. The feedback below comes from merging 2 responses from Chatgpt to 2 near-identical prompts (and 1 other response to try to clarify some info provided ) merged with some minor revision by me, with my comments.

   I chose this “best smartphones 2025” prompt as it demands “up to date” information: it is a prompt that can not be resolved reliably from the information in the model's knowledge base alone (as the cut-off point for the model's knowledge base is mid-2024)

The stages in response formation Chatgpt stated were as follows:

Stages 1-3. Prompt interpretation / Vector representation generation / Contextual focus mechanism.

  Chatgpt stated the response generation starts with the prompt (“best smartphones 2025”) which is tokenized and passed through the model's transformer layers and generates candidate brands / models / specifications using it's internal knowledge (a.k.a. the “corpus”) from it's training data up to mid-2024. 

   The model then maps each token in the prompt to dense vectors in high-dimensional space. Here's what activates:

   Chatgpt reported that ALL the info at this stage came entirely from the model’s pretrained corpus, up to a 2024 cutoff. 

     Here is the knowledge in this response that came from training data up to mid-2024 alone below: 

• Understanding that flagship phones are typically released in Q1 (Samsung, OnePlus, Xiaomi) or Q3 (Apple)
  

• Phone series evolution: iPhone naming convention (e.g. iPhone 17 follows 16), Samsung S-series (S25 after S24)*
  

• Common flagship specs: e.g. high refresh rate OLEDs, multi-lens cameras, Snapdragon / Exynos / A-series chips
  

• AI feature trends (e.g. multimodal assistants, fast charging over 100W etc )
  

• Typical vendors for foldables (Samsung leads), and Xiaomi’s camera collaboration with Leica
  
  

   This information Chatgpt stated were retrieved through high-level embeddings based on semantic similarity between the prompt (“best smartphones 2025”) and vectors representing past smartphone rankings and product specs. It is worthwhile re-iterating Chatgpt stated that “No access to real-time 2025 device details was available in pre-training data” - all of the information above was from the corpus or inferred (the annual release of IPhones with a naming convention e.g. 13 is followed by 14).

Stage 4. Web tool triggered

   Chatgpt stated that it then “issued live web searches to retrieve high-precision sources to confirm whether these models exist”*, their specs, and any recent reviews or announcements.. Those results are ranked for topical match and credibility.

    *N.B. Chatgpt in it's reponse above stated it “generates candidate brands / models and specifications using it's internal knowledge (a.k.a. the “corpus”) from it's training data up to mid-2024”. From what I see sometimes in Chatgpt you can see responses based on naming conventions (e.g. iPhone 17 follows 16) for models / products that may not have yet been released or were never released (in responses where the Web tool wasn't triggered to “fact check”); thus as Chatgpt fed back in this instance it “issued live web searches to retrieve high-precision sources to confirm whether these models exist”. Essentially the tool “predicted” the new product release based on naming convention / release schedule information in the corpus and then checked its "facts" via the web .

   Live search was used to pull up-to-date 2025 model names, features, and rankings. Here's the exact web search pipeline reported: (note I have broken some URLs)

Search 1 (Google): (my notes: I'm not sure if this means all searches were done using Google as it was the only search labelled as such)

    Query: “best smartphones 2025 site:techadvisor.com OR site:tomsguide.com OR site:wired.com” (my notes: note the use of “notable” tech / review sites in this search)

• Source used: TechAdvisor 2025 Best Phones Guide https://www.techadvisor .com/article/724318/best-phone.html

  • Chatgpt stated it “Verified Samsung Galaxy S25 Ultra, OnePlus 13, Xiaomi 15 Ultra specs and rankings” data using this source.
    

Search 2:

Query: “iPhone 17 Pro Max specs 2025”

• Source used: https://www.macrumors .com/roundup/iphone-17/

  • Chatgpt stated it “Verified [the] A19 chip, 120 Hz ProMotion display, iOS 18 feature set” details using this source
    

Search 3:

Query: “Galaxy Z Fold 7 specs review 2025”

• Source used: Tom’s Guide comparison of the Z Fold 7 & Galaxy S25 https://www.tomsguide .com/phones/samsung-phones/i-put-the-galaxy-z-fold-7-vs-galaxy-s25-ultra-through-a-7-round-face-off-heres-the-winner

  • Chatgpt stated it “Verified [the] foldable screen size, multitasking UI, and S25 Ultra comparison” using this source
    

Search 4:

Query: “Xiaomi 15 Ultra review 2025”

• Sources used: https://www.techadvisor .com/article/724318/best-phone.htmland & https://en.wikipedia.org/wiki/Xiaomi_15

•  Chatgpt stated this search “allowed the response to include specific model names, specs (e.g., Snapdragon 8 Gen 4 vs A19), camera details, display sizes, and expert opinions” using this source.
  
  

    Then, Chatgpt then merged the corpus information with the above information from the Web tool —relying on live data for specifics and using the trained patterns for presentation, organization, and comparison logic.

Stage 5 . Response generation:

    Chatgpt used the latest factual information with pretrained semantic structures to output a comparison-oriented list, emphasizing criteria commonly used in smartphone rankings (camera, chip, battery, UI, price etc). 

   I did the same in Google Gemini too.