SAST THU CST SAST Homepage ↗

Leaderboard

Five models and three baselines on the same 212 tasks. Baselines decide every 15 minutes; agents decide hourly. Both may add up to six event-triggered decisions per local day. Each episode lasts 72 hours, meals and activity are never reported, and higher is better.

  1. Claude Opus 5.5 73.54
  2. Simulated oref1 (15 min) 71.19
  3. GPT 6.1 Sol 70.58
  4. DeepSeek 4.1 Flash 68.61
  5. Kimi K3.1 Pro 67.15
  6. GLM 5.3 Flash 66.25
  7. Guideline rules (15 min) 63.57
  8. Standard care (15 min) 61.16

How COGI is scored

Every score comes from true glucose every five minutes, which the agent never sees.

  • 50%

    Time in range

    True glucose inside 4–8 mmol/L.

  • 35%

    Hypoglycemia

    Falls to zero at 15% of time below 70 mg/dL.

  • 15%

    Variability

    Falls to zero at a standard deviation of 108 mg/dL.

All controllers

Controller COGI Time in range Hypoglycemia Variability
Claude Opus 5.5 73.54 54.58 97.29 81.28
Simulated oref1 (15 min) 71.19 51.62 96.45 77.48
GPT 6.1 Sol 70.58 49.07 97.93 78.42
DeepSeek 4.1 Flash 68.61 44.95 97.25 80.64
Kimi K3.1 Pro 67.15 43.58 96.65 76.90
GLM 5.3 Flash 66.25 41.00 96.80 79.17
Guideline rules (15 min) 63.57 38.35 95.04 74.21
Standard care (15 min) 61.16 33.82 94.54 74.42

Every task, every model

Each dot is one task and the bar marks the mean. Claude Opus 5.5 posts the best score on 141 of 212 tasks.

40 50 60 70 80 90 100 Claude Opus 5.5 GPT 6.1 Sol DeepSeek 4.1 Flash Kimi K3.1 Pro GLM 5.3 Flash

By scenario

A failing pump is the hardest problem for every model: the sensor shows the rise, but not its cause.

Scenario group Claude Opus 5.5GPT 6.1 SolDeepSeek 4.1 FlashKimi K3.1 ProGLM 5.3 Flash
Meal timing and absorption 77.2 72.2 71.8 70.3 69.5
Insulin from earlier doses 78.3 74.1 73.0 72.6 70.1
Exercise 81.3 75.9 76.3 72.0 74.2
Changing insulin needs 79.5 75.1 75.4 72.1 73.2
Pump under-delivery 52.2 55.4 46.6 49.6 46.7
Sensor errors 71.4 68.1 66.4 65.2 64.5
Illness and recovery 74.3 72.9 70.0 67.6 64.9

By person

Virtual person Claude Opus 5.5GPT 6.1 SolDeepSeek 4.1 FlashKimi K3.1 ProGLM 5.3 Flash
Children, 9 y 70.6 67.4 65.1 63.4 62.5
Adolescents, 15 y 71.7 69.4 66.8 66.3 64.3
Adults, 39 y 77.3 73.8 73.0 70.6 70.5
Older adults, 70 y 74.6 71.7 69.5 68.3 67.7
← Back to DiabAgent