That double negative “don’t disagree” connotates that “I don’t have an argument with what you’re saying, but I don’t really like it either”. The speaker might be trying to sound less directly opposing, but those words usually don’t land well.

A better way would be to be more direct and say what it is concerning you about the idea. “I’m concerned about [a specific detail of the idea].”