I spent 5 weeks building an open-source multi-agent orchestrator. The hard part wasn't the agents — it was the memory.