Yes, there is usually a fee for crossing the border with a rental car. You will find information on this in the rental conditions of your preferred offer.
The amount varies from provider to provider. Sometimes the deposit also increases when you cross the border. So check the rental conditions for your offer carefully. It may be better to book two rental cars in two countries rather than crossing both countries with one vehicle.
Especially if you are planning a long trip by rental car, it makes sense to do it in several stages by changing vehicles. On the one hand, you save the additional fees for crossing the border, and on the other hand, most car rental companies have set a maximum rental period between a maximum of 28 days (in Europe) and a maximum of 6 weeks (in the USA).