I see you are there already - 500 ms latency through the system suggests something else is not configured the best way. In general I suggest running the SDR hardware at the highest possible sample rate for dynamic range reasons. But, in this case this might result in reduced latency as the buffers would fill quicker and get forwarded to the SDR program. In any case that delay you seem to face would make CW conversations a little awkward and contesting might suffer from the delay between antenna and headset. I wonder if you are seeing a delay within your transmitter where it's internal signal generation does not end promptly.

I missed what you are using for SDR hardware. That, too, may make a significant difference.

Out of addled curiosity are you associated with the ARRL? If so, welcome to the 21st century. I gave up hope for the ARRL with regards technology beyond what you can do with empty-state devices back in the 80s, 90s, and aughties. That last one seems to have stuck. I got tired of the means my which a signal is generated being magic that transforms a signal from one designation to another. (That is an FCC problem, too, I realize. ARRL's guidance to the FCC seems lacking.)

{o.o}