Is there a way to restrict my recording from download by the viewers?

Yes, if they are cloud recordings.  You can follow the steps below to arrange the setting:
  1. Login HKU Zoom.
  2. Click Recordings on the left menu. Under Cloud Recordings, click the Share button for the recording you wish to disable the download
  3. Show Zoom Recording page settings
  4. Check to make sure the option Viewers can download is off.  Turn on the Share this recording and Only authenticated users can view options if you want your video be viewed by HKU staff and students only (i.e. requires HKU Portal login to view).
  5. Show Zoom Recording page menu