A serial connection is made with a serial port (COM port) on your pc. There are usb-serial cables for this. The usb input is routed to a com port on your pc.
However if you really have written an incompatible uboot to flash than you can forget that. You will need special tools as jtag interface and an uboot image for this device.
In case it was a 5big2 kirkwood based device there would be a backdoor by loading an uboot image via uart (serial port) but I don't know if this can work in a 5big1 which has an orion processor.
Example see here
https://forum.doozan.com/read.php?3,7852,7852 I am afraid you have turned your nas into a brick if you really have written a non compatible uboot image.
But how exactly did you write a new uboot image? Its not so easy to write uboot to flash.
Or did you try to write new firmware to a disk?