Why include CLS token in the text embedding #40

New Issue

2023-10-10T11:38:41+08:00

Trainingzy commented

2023-10-10 11:38:41 +08:00

(Migrated from github.com)

Thanks for your great work!

I notice that you include both hidden state and CLS token in stable diffusion text embeddings here. However, in stable diffusion 1.5, the text embedding does not include the CLS token.

I also checked stable dreamfusion. It also not use CLS token here.

May I know why you include CLS token here?

Thanks for your great work! I notice that you include both hidden state and CLS token in stable diffusion text embeddings [here](https://github.com/guochengqian/Magic123/blob/7f051875e48988dc624b2bd8425aea370c093588/nerf/utils.py#L351C80-L351C80). However, in stable diffusion 1.5, the text embedding does not include the CLS token. I also checked stable dreamfusion. It also not use CLS token [here](https://github.com/ashawkey/stable-dreamfusion/blob/5550b91862a3af7842bb04875b7f1211e5095a63/nerf/utils.py#L359). May I know why you include CLS token here?

Wakeelfahmed commented

2023-10-13 15:32:07 +08:00

(Migrated from github.com)

are u able to use the pretrained model? I am unable to set up the model/environment.
Would u please help?

are u able to use the pretrained model? I am unable to set up the model/environment. Would u please help?

guochengqian commented

2023-10-13 16:48:01 +08:00

(Migrated from github.com)

we also do not use CLS token:
7f051875e4/nerf/utils.py (L732)
text_z_sds = text_z[:, :-1]

I think I keep the CLS token for the CLIP. But we do not use that eventually.

we also do not use CLS token: https://github.com/guochengqian/Magic123/blob/7f051875e48988dc624b2bd8425aea370c093588/nerf/utils.py#L732 text_z_sds = text_z[:, :-1] I think I keep the CLS token for the CLIP. But we do not use that eventually.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dearsky/Magic123#40