Setting up a Caravan network is needless busywork. It's also a complete interface cludge that's just really, really unpleasant to deal with.

Think it's fine in your little empire? Try ensuring every one of your cities is connected on a larger map, in the midgame, on a higher difficulty, when cities are rapidly changing hands and routes are getting broken or ravaged by monsters. It sucks.

I don't have any issue with attacking on-map caravans (though again, visuals get in the way of this, unless you zoom out to cloth view, caravans are not easy to spot), but the process of building, evicting, and targeting every caravan - then trying to confirm if it arrived at the right city, then discovering 20 turns later that you have a caravan idling in a city, or arrived at the wrong city - not one bit of it is fun or interesting.

Handle it through a special UI panel, handle it through a Trade window on the kingdom interface, handle it through a Trade tab on the city, whatever, just don't handle it through the unit movement ui.