I have not managed to hit a limit and I've had some monstrous trees. However, I've discovered that branching text trees horribly bog down the Foundry. I had several trees with at least 30 boxes, and the one with the most had 56. However, I had to remove them with the intention of putting them back in later because the Foundry became unusable. It took more than 30 seconds to place an item or adjust it, so I simply couldn't create any more maps.
Based on my experience you'll hit performance limits long before you hit hard limits. Granted, this could just be my machine. Maybe it's because I only have 2 GBs of memory, or a 2.5 GHz x2 processor. Regardless, I've learned to basically create my entire mission first and put all the dialogue in last. It's kind of a pain in the butt, but it's the only thing that will even remotely work for me.
It's unlikely that the slowdown has anything to do with our client-side machines; it's likely something server-side which hasn't been optimised yet. A check on all active map items for example, which iterates on every change.
Well, I do the text last. I build up all of my maps and then put the dialogue in after I'm done. that allows me to run through the map several times to make sure the elements work without having to deal with my own rambling.
I have a pretty big tree at the end of the latest mission. I wanted to simulate a computer file system.