Local LLM (llama.cpp / Ollama) benchmarks

Tokens a second generating text from an open model run locally with llama.cpp or a tool built on it (Ollama, LM Studio). Generation speed follows memory bandwidth; what fits follows memory size.

7 leaderboards · tok/s, higher is better

Llama 2 7B (MLPerf Client 0.5), Generation

5 products · tok/s · main source Tom's Hardware
  1. 1RX 9070 XT113
  2. 2RX 9070110
  3. 3Arc B58080.5
  4. 4Arc B57070
  5. 5RX 9060 XT 16GB67
Sources and dates
  • RX 9070 XT: Tom's Hardware, Mar 5, 2025 (Ryzen 7 9800X3D, 32GB DDR5-6000, ASRock X670E Taichi (Tom's Hardware 2025 GPU testbed))
  • RX 9070: Tom's Hardware, Mar 5, 2025 (Ryzen 7 9800X3D, 32GB DDR5-6000, ASRock X670E Taichi (Tom's Hardware 2025 GPU testbed))
  • Arc B580: Tom's Hardware, Dec 13, 2024 (Tom's Hardware GPU test PC (the review used a Ryzen 7 9800X3D and a Core i9-13900K system))
  • Arc B570: Tom's Hardware, Jan 16, 2025 (Ryzen 7 9800X3D, 32GB DDR5-6000, ASRock X670E Taichi (Tom's Hardware 2025 GPU testbed))
  • RX 9060 XT 16GB: Tom's Hardware, Jun 10, 2025 (Ryzen 7 9800X3D, 32GB DDR5-6000, ASRock X670E Taichi (Tom's Hardware 2025 GPU testbed))

Main source: Tom's Hardware.

Llama 2 7B F16, Generation

4 products · tok/s · main source llama.cpp community benchmarks (GitHub)
  1. 1Mac Studio M5 Ultra73.6
  2. 2Mac Studio M5 Max37.1
  3. 3MacBook Pro 14 M5 Pro21.4
  4. 4iMac M47.43
Sources and dates

Main source: llama.cpp community benchmarks (GitHub).

Llama 2 7B Q4_0, Generation

13 products · tok/s · main source llama.cpp community benchmarks (GitHub)
  1. 1Mac Studio M5 Ultra179
  2. 2Mac Studio M5 Max120
  3. 3MacBook Pro 14 M5 Max120
  4. 4MacBook Pro 16 M5 Max120
  5. 5Mac mini M5 Pro67.6
  6. 6MacBook Pro 14 M5 Pro67.6
  7. 7MacBook Pro 16 M5 Pro66.3
  8. 8Mac mini M4 Pro49.6
  9. 9Mac mini M635.7
  10. 10MacBook Air 15 M531.9
  11. 11MacBook Pro 14 M531.9
  12. 12Mac mini M424.1
  13. 13iMac M424.1
Sources and dates

Main source: llama.cpp community benchmarks (GitHub).

Llama 2 7B Q4_0, Prompt processing

13 products · tok/s · main source llama.cpp community benchmarks (GitHub)
  1. 1Mac Studio M5 Ultra4,945
  2. 2Mac Studio M5 Max3,220
  3. 3MacBook Pro 14 M5 Max3,220
  4. 4MacBook Pro 16 M5 Max3,220
  5. 5MacBook Pro 16 M5 Pro1,621
  6. 6Mac mini M5 Pro1,340
  7. 7MacBook Pro 14 M5 Pro1,340
  8. 8Mac mini M6989
  9. 9MacBook Air 15 M5723
  10. 10MacBook Pro 14 M5723
  11. 11Mac mini M4 Pro364
  12. 12Mac mini M4221
  13. 13iMac M4221
Sources and dates

Main source: llama.cpp community benchmarks (GitHub).

pp512 Llama 2 7B Q4_0, Prompt processing

11 products · tok/s · main source llama.cpp community benchmarks (GitHub)
  1. 1RTX 509014,073
  2. 2RTX 50808,297
  3. 3RTX 5070 Ti6,952
  4. 4RTX 50705,185
  5. 5RX 9070 XT5,036
  6. 6RTX 5060 Ti 16GB3,737
  7. 7RTX 50603,269
  8. 8RX 90703,164
  9. 9RX 9060 XT 16GB2,845
  10. 10Arc B570914
  11. 11Arc B580621
Sources and dates

Main source: llama.cpp community benchmarks (GitHub).

tg128 Llama 2 7B Q4_0, Generation

11 products · tok/s · main source llama.cpp community benchmarks (GitHub)
  1. 1RTX 5090290
  2. 2RTX 5080182
  3. 3RTX 5070 Ti177
  4. 4RX 9070 XT137
  5. 5RTX 5070128
  6. 6RX 9070120
  7. 7RTX 506096.7
  8. 8RTX 5060 Ti 16GB90.9
  9. 9RX 9060 XT 16GB71.1
  10. 10Arc B58070.1
  11. 11Arc B57049.6
Sources and dates

Main source: llama.cpp community benchmarks (GitHub).

Llama 2 7B Q8_0, Generation

8 products · tok/s · main source llama.cpp community benchmarks (GitHub)
  1. 1Mac Studio M5 Max72.4
  2. 2Mac mini M5 Pro38.9
  3. 3MacBook Pro 14 M5 Pro38.9
  4. 4Mac mini M4 Pro30.5
  5. 5Mac mini M621
  6. 6MacBook Air 15 M518.4
  7. 7Mac mini M413.5
  8. 8iMac M413.5
Sources and dates

Main source: llama.cpp community benchmarks (GitHub).