Attention Mechanism is Very Slow

[prajdabre]

Hi,
Your attention mechanism is quite slow. Since you compute the linear projections (aw and bw) each time although they do not change, the time is almost quadratic.

I have implemented a faster version of attention which does a lot of precomputation and would like to push it as soon as I am done testing.

Regards.

[odashi]

Thank you, I had overlooked this issue and I think it is no problem to fix about that calculation.

Regards,

[prajdabre]

Hi,
I have implemented the fast version of attention and have tested it.
I will push it tomorrow or so.