I'm running HAOS on a mini x86 PC with localai.io running on a local server with an Intel A750. The model I'm currently using is qwen3:4b-instruct-2507-q4_K_M which should support tool use.

I've tried other models and had similar results.

The debug spew for the above interaction is below. Of note, the data field of intent_output is empty.

Is this something I need to configure on the localai.io side? Something with the yaml file? Or is HA not catching the intents coming from localai?

stage: done
run:
  pipeline: 01m4qn8aamt08pa8q3gprs8g9y
  language: en
  conversation_id: 01M4SQBP7364Q234NHBDAVQF4V
  runner_data:
    stt_binary_handler_id: null
    timeout: 300
events:
  - type: run-start
    data:
      pipeline: 01m4qn8aamt08pa8q3gprs8g9y
      language: en
      conversation_id: 01M4SQBP7364Q234NHBDAVQF4V
      runner_data:
        stt_binary_handler_id: null
        timeout: 300
    timestamp: '2026-10-13T04:12:03.684196+00:00'
  - type: intent-start
    data:
      engine: conversation.ollama_conversation
      language: en
      intent_input: turn off the kitchen lights
      conversation_id: 01M4SQBP7364Q234NHBDAVQF4V
      device_id: null
      satellite_id: null
      prefer_local_intents: false
    timestamp: '2026-10-13T04:12:03.684286+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        role: assistant
        content: ''
    timestamp: '2026-10-13T04:12:03.834864+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: intent
    timestamp: '2026-10-13T04:12:03.835315+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: __
    timestamp: '2026-10-13T04:12:03.888268+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: H
    timestamp: '2026-10-13T04:12:03.941413+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ass
    timestamp: '2026-10-13T04:12:03.997781+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: Turn
    timestamp: '2026-10-13T04:12:04.052061+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: 'Off'
    timestamp: '2026-10-13T04:12:04.108197+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ("
    timestamp: '2026-10-13T04:12:04.160317+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: light
    timestamp: '2026-10-13T04:12:04.214299+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: '",'
    timestamp: '2026-10-13T04:12:04.267968+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' "'
    timestamp: '2026-10-13T04:12:04.320906+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: Kitchen
    timestamp: '2026-10-13T04:12:04.376033+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: '")'
    timestamp: '2026-10-13T04:12:04.432053+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ''
    timestamp: '2026-10-13T04:12:04.485825+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ''
    timestamp: '2026-10-13T04:12:04.486312+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ''
    timestamp: '2026-10-13T04:12:04.486666+00:00'
  - type: intent-end
    data:
      processed_locally: false
      intent_output:
        response:
          speech:
            plain:
              speech: intent__HassTurnOff("light", "Kitchen")
              extra_data: null
          card: {}
          language: en
          response_type: action_done
          data:
            success: []
            failed: []
        conversation_id: 01M4SQBP7364Q234NHBDAVQF4V
        continue_conversation: false
    timestamp: '2026-10-13T04:12:04.488028+00:00'
  - type: run-end
    data: null
    timestamp: '2026-10-13T04:12:04.488136+00:00'
started: 2026-10-13T04:12:03.684Z
intent:
  engine: conversation.ollama_conversation
  language: en
  intent_input: turn off the kitchen lights
  conversation_id: 01M4SQBP7364Q234NHBDAVQF4V
  device_id: null
  satellite_id: null
  prefer_local_intents: false
  done: true
  processed_locally: false
  intent_output:
    response:
      speech:
        plain:
          speech: intent__HassTurnOff("light", "Kitchen")
          extra_data: null
      card: {}
      language: en
      response_type: action_done
      data:
        success: []
        failed: []
    conversation_id: 01M4SQBP7364Q234NHBDAVQF4V
    continue_conversation: false
finished: 2026-10-13T04:12:04.488Z

Yes. I even tried dropping down to just two entities exposed, and nothing changed.

What else can you use LLM for? I saw Smartlife enshittified by adding AI prompt bar in every page and i can't think of anything to use it, it's simpler if i just program the automation myself, or click the button.

While I agree that using LLM chats for deterministic tasks introduces probability where none is needed it seems like a good way to check that the connection works as wanted. The success is immediately visible and if it does something weird it's just a lamp it can control.

I am confused. The default whisper + HomeAssistant assistant + piper pipeline works completely fine for day to day tasks, turning lights on and off, etc...

Why would OP try to host a local LLM for the same basic intent recognition that home assistant does pretty well already?

I'm not a fan of how specific it needs you to be with wording. Took me too long to realize "turn lamp to 50%" had to be "turn lamp brightness to 50%" The former gives you nothing.

Also, it's fun!

Figured it out. I switched from the ollama service to llama.CPP and that seems to have fixed it.

It can have it's use cases.
Family members don't have to be as precise with correct sentences for example.
If you want to use it with music assistant a simple LLM works better in my experience then trying to do this with intent.
Can you do all this with the local intent recognition? Absolutely, but you will experience edge cases and it takes a while to get dialled in correctly.

An other big plus, LLMs often automatically support multi language. This is big for a lot of European users in my family we already speak 4 different languages and various dialects and now no matter who visits or stays over, they can use the voice control in native langue.

Personally, I use both. If the intent recognition doesn't catch it, the LLM will. So for me it is just an extra layer that protects me from angry family members :)

Setup:

• Home assistant voice
• Ollama running on a GTX1660Super with granite4.2:3b
• Whisper small running on same machine.

It runs decently well, especially for tool use tasks.

The multi language is great, true, but that starts with good multi language STT, which in my experience, Flemish Dutch has like a 70-80% error rate because they are all trained on Netherlands dutch, and that isn't even with a west Flemish accent lol.