By default Caldera Forms adds a shortcode insert button for the Caldera Forms shortcode to the post editor. If you would like to remove this button, you can unregister that button:
You should also remove the JavaScript and CSS for the shortcode preview and button: