There's 2 reason why speed suffers... here's why: (I'll assume copying a file to the server for my examples, but going the other way has the same limitations)
When copying a file the sharing protocol(as far as I know all of them work this way) requires extra communication for each file. Here's basically what the protocol looks like. Each step won't occur until the previous step is complete.
1. Workstation -> Server : "I want a new file called document.doc"
2. Server -> Workstation : "Your file is created.. send me any contents you want" (The server verifies appropriate permissions and creates the file)
3. Workstation -> Server : File contents sent. (The server does send acknowledgement packets as the file transfers too)
4. Workstation -> Server : "File contents completely sent. Please close the file"
5. Server -> Workstation : "File is closed and saved to the hard drive" (The server MUST complete all applicable writes from the write cache before sending this command. This is often called a sync write.)*
6. Workstation will then delete the file locally or begin with #1 if only copying.
Overall, at each step you are waiting for the other machine to complete their step.
Now, also consider the fact that as each read or write is performed, if you are using magnetic media you will also be adding in seek times for the hard disk heads. Typically, if you have a seek time of 5ms you can expect an absolute maximum of 200 writes per second without any consideration of any other latency. Now consider that you are also executing 3 independent writes for each file... the file system creating the file, the file contents being written, and the file system closing the file. So now, you are really talking about a maximum of, at best, about 65 files per second. Add in the latency of the protocol plus the fact that the workstation will also have it's own 5ms of seek times and you can easily find yourself with only 10-30 files per second. If each of those files is only a few kb, then you can expect only 10-30 times the file size for total transfer rate.
There's no easy way to beat this. Even when one side has an SSD(thereby removing the 5ms penalty for seeks) the other side will still be quite busy trying to complete those seeks.
* - If the amount of data that needs to be written is less than 64 kilobytes and you have a ZIL for the zpool(preferably on an SSD) then the sync write will happen to the ZIL instead of your pool and you will save a small amount of time. However, your ZIL will still have to be process at some point in the future(typically within 2-15 seconds) so your zpool will still end up busy at some point in the very near future. This could potentially slow down future writes too. There is a way to disable ZIL sync writes without a ZIL, but that is extremely dangerous for your data and can result in corruption of the zpool and loss of data. I don't remember the exact tweak to disable sync writes, but if I did know it I don't know if I'd share it because it is almost irresponsible to use it in most situations. For small file transfers it won't significantly help anyway.
Hopefully this all makes sense. :P