o …ðÔi/ã@s¨ddlmZddlmZmZmZddlZddlmZdejdejdejfdd „Z Gd d „d ej ƒZ Gd d „d eƒZ Gdd„dej ƒZ Gdd„de ƒZGdd„de ƒZdS)é)Ú namedtuple)Ú NamedTupleÚOptionalÚTupleN)ÚnnÚxÚmeanÚtxcCsŒ|jdkr||}||j}|S|jdkr:|| dddd¡}|jg|j¢d‘d‘RŽ}tjjj||dddd�}|Std|j›d|j›�ƒ‚) Nééééÿÿÿÿr)ÚweightÚbiasÚstrideÚpaddingúUnsupported input dimension: z , shape: ) ÚndimÚTÚreshapeÚshapeÚtorchrÚ functionalÚconv2dÚ ValueError)rrr Úkernel©rú?/home/zhangtao/hf_model/vectorllm_hf_0407/feature_normalizer.pyÚ _run_kernels   úÿrcsDeZdZejfdedejf‡fdd„ Zdejdejfdd„Z ‡Z S) ÚFeatureNormalizerÚ embed_dimÚdtypecs:tƒ ¡| dtj||d�¡| dtj||d�¡dS)Nr©r!r )ÚsuperÚ__init__Úregister_bufferrÚzerosÚeye)Úselfr r!©Ú __class__rrr$s zFeatureNormalizer.__init__rÚreturncCst||j|jƒ}|S©N)rrr )r(rrrrÚforward"szFeatureNormalizer.forward) Ú__name__Ú __module__Ú __qualname__rÚfloat32Úintr!r$ÚTensorr-Ú __classcell__rrr)rrsrc@s"eZdZUejed<ejed<dS)ÚInterFeatStateÚyÚalphaN)r.r/r0rr3Ú__annotations__rrrrr5's r5c @s2eZdZd dejdededeedef dd„ZdS) Ú!IntermediateFeatureNormalizerBaseNrÚindexÚ rot_indexÚskipr+cCstƒ‚r,)ÚNotImplementedError©r(rr:r;r<rrrr--sz)IntermediateFeatureNormalizerBase.forward©NN) r.r/r0rr3r2rr5r-rrrrr9,s*r9c sreZdZdejfdedededejf‡fdd„ Zdd ej d ed ed e ed e f dd„Z d ed ej fdd„Z ‡ZS)ÚIntermediateFeatureNormalizerFÚnum_intermediatesr Ú rot_per_layerr!csrtƒ ¡| dtj||d�¡tj||d�}|r#| d¡ |dd¡}| d| ¡¡| dtj |||d�¡dS)NÚalphasr"rr ÚrotationÚmeans) r#r$r%rÚonesr'Ú unsqueezeÚrepeatÚ contiguousr&)r(rAr rBr!Úrotr)rrr$2s z&IntermediateFeatureNormalizer.__init__Nrr:r;r<r+c Cs<|dur|}|r&|jdksJdƒ‚|dd…d|…f|dd…|d…f}}| |¡}t||j||ƒ}|j|}|rat tj||j|j d�|d  |j d¡g¡  ddd¡}tj||gdd�}n8|jdkrv|  ddd¡  d|j dd¡}n#|jdkr‘|  dddd¡j ddg|j dd…¢RŽ}nt d |j›�ƒ‚t||ƒS) Nr zHCannot use the `skip` parameter when the `x` tensor isn't 3-dimensional.©r!Údevicer r )Údimr ér)rÚ _get_rotationrrErCrÚcatrFr!rLÚexpandrrrr5) r(rr:r;r<ÚprefixrDr6r7rrrr-=s,*  þ ý   , z%IntermediateFeatureNormalizer.forwardcCs|jjdkr |jS|j|S)NrN)rDr)r(r;rrrrOYs  z+IntermediateFeatureNormalizer._get_rotationr?)r.r/r0rr1r2Úboolr!r$r3rr5r-rOr4rrr)rr@1s(& r@c speZdZeƒZdejdejf‡fdd„ Ze dejdejfdd„ƒZ ddej d e d e d e e d ef d d„Z‡ZS)Ú!NullIntermediateFeatureNormalizerr!rLcs&tƒ ¡| dtjd||d�¡dS)Nr7r rK)r#r$r%rÚtensor)r(r!rLr)rrr$bs z*NullIntermediateFeatureNormalizer.__init__cCs6tj ||fd¡}|durt||ƒ}|tj||f<|Sr,)rTÚ instancesÚget)r!rLÚinstancerrrÚ get_instancefs  z.NullIntermediateFeatureNormalizer.get_instanceNrr:r;r<r+cCs t||jƒSr,)r5r7r>rrrr-ns z)NullIntermediateFeatureNormalizer.forwardr?)r.r/r0ÚdictrVrr!rLr$Ú staticmethodrYr3r2rr5r-r4rrr)rrT_s .rT)Ú collectionsrÚtypingrrrrrr3rÚModulerr5r9r@rTrrrrÚs    .