The vacuum for the exhaust flap comes from a connection at the back of the intake manifold. If you're not going to use the vacuum flap (since you remove your muffler), just cap the port on the back of the intake manifold.

#1, just remove the vacuum line and plug the port. The other port right under #1 is for the Secondary Air Pump normally, but I'm unsure if SAP is installed on your specific car.

(Not my pic, just copy-n-paste from another post. Ignore #2 for this discussion.)

 

It looks like it been for a while without vacuum hose. Would I felt how car behaves?
Also on realoem picture I have provided, it looks like the hose isn't connected and shows plug and zip wire. That sort of made me question if there was a vacuum hose at all.
Other thing how difficult is to get to that port? Thank you

 

If this vacuum port was left open then you would get a lean condition as mentioned above, and if it's lean enough that will result in P0174/P0171 codes.