SAM 3 Segmentation Agent Now in ComfyUI

The article discusses the integration of the SAM 3 segmentation model into the ComfyUI tool, allowing for more advanced character segmentation in Stable Diffusion images.

Details

The article discusses the integration of the SAM 3 segmentation model into the ComfyUI tool, which allows for more advanced character segmentation in Stable Diffusion images. The author explains that while SAM 3 is great at segmenting general concepts, it struggles with character-specific descriptions like 'the fourth woman from the left holding a suitcase'. To address this, the author has adapted the SAM 3 Agent example notebook into a ComfyUI node that works with both local GGUF VLMs and through OpenRouter. The agentic process involves the agent analyzing the base image and character description prompt, choosing appropriate simple noun phrases for segmentation, and iterating until satisfactory masks are found. However, the author notes that this agentic process is often slower and less accurate than purpose-trained solutions like Grounded SAM and Sa2VA. The author suggests future improvements, such as refining the system prompt, using Grounded SAM or Sa2VA with the agentic loop, and exploring bounding box/pointing VLMs.