Xaro Sent the Sorrowful Man

There is no conclusive proof, and probably never will be, but a variety of hints suggest Xaro Xhoan Daxos hired the Sorrowful Man who tried to kill Daenerys with a manticore at the docks. His purpose: to acquire her dragons.

His plan, in the books, seems to have gone like this:

1. Welcome Daenerys and her dragons into his mansion. The concentric walls of the city suggest going inward to be entering a trap.

2. Establish a public expectation that the dragons live there because on display there. The sideshow would also convince Daenerys she gained by his generosity and was still free.

3. Convince Daenerys to bankrupt herself spending the dragon side show proceeds trying to bribe the Pureborn. By the way he reacts when she says she wants her bribes back because the men she'd bribed did not speak for her, it seems Xaro has misappropriated the funds and did not bribe the Pureborn at all. Or, he knew they’d never speak for her or offer a beverage. Xaro clearly sent Daenerys to the Pureborn to be bankrupted and disappointed.

4. With Daenerys and her dragons in his walls, and her without a way to pay for ships, Xaro then offers the carrot or the stick: she can marry him and part with at least one dragon legally or refuse him and, “breaking his heart,” be asked to depart without the means to actually do so. This does happen and Xaro begins to send increasingly frosty messages about no longer wishing to feed her people. It may be this is designed to convince Daenerys to marry him or trade a dragon for ships, but it also serves to set up that he can evict her people frostily were she to suddenly die.

5. Send a Sorrowful Man to kill her. Had the Sorrowful Man with the manticore at the docks succeeded in killing Daenerys, Xaro would have continued to press her people to leave, but not allowed them to take the dragons. As you point out, Dany only had a few men. Even if a few Dothraki were willing to die to guard Dany's dragons, they'd be easily overpowered. Or, could frankly keep guarding them. When the dragons entered Xaro's home, they became his all but legally.